Nov 03 19:30:51.995875 ip-10-0-133-204 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:30:52.434429 ip-10-0-133-204 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:30:52.434429 ip-10-0-133-204 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:30:52.434429 ip-10-0-133-204 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:30:52.434429 ip-10-0-133-204 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:30:52.438605 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438514 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.438605 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438606 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438610 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438615 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438618 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438621 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438623 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438626 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438628 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438631 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438634 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438636 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438639 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438642 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438644 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438647 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438649 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438652 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438655 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438657 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438660 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.438678 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438662 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438665 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438668 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438670 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438674 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438676 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438679 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438682 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438684 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438687 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438690 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438693 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438696 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438698 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438701 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438703 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438706 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438708 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438711 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438713 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.439141 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438716 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438718 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438720 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438723 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438726 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438728 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438730 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438733 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438735 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438738 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438740 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438742 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438745 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438748 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438752 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438754 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438757 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438759 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438762 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438765 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.439638 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438767 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438770 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438772 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438775 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438777 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438780 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438782 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438785 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438787 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438790 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438792 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438795 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438798 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438800 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438803 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438810 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438813 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438820 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438823 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438826 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.440134 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438829 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438831 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438834 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438837 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438840 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438842 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438845 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438847 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438850 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438853 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438855 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438858 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438860 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438863 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438868 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438872 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438875 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438878 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438881 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.440621 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438884 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438886 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438889 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438891 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438896 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.438900 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439641 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439649 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439652 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439655 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439658 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439661 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439664 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439667 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439670 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439680 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439683 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439686 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439688 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.441094 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439691 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439694 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439696 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439699 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439702 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439704 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439707 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439710 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439712 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439715 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439717 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439721 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439723 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439726 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439728 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439731 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439733 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439736 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439739 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.441594 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439741 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439744 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439746 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439749 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439752 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439755 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439757 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439760 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439763 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439766 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439768 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439771 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439773 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439776 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439778 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439781 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439784 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439786 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439788 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439791 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.442050 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439794 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439796 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439799 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439801 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439804 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439806 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439809 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439811 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439814 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439816 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439819 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439821 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439824 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439827 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439829 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439832 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439834 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439837 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439840 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439843 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.442524 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439845 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439849 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439852 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439855 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439858 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439860 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439863 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439866 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439868 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439871 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439873 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439876 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439879 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439881 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439884 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439886 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439889 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439891 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439894 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439896 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.443053 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439899 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439901 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439904 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439906 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439909 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439911 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439914 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439917 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439921 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439925 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439929 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439932 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439935 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.439940 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440406 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440412 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440416 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440418 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.443516 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440421 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440424 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440426 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440429 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440431 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440434 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440436 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440439 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440442 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440444 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440447 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440450 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440452 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440454 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440457 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440460 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440463 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440465 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440468 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440470 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.443971 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440473 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440475 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440478 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440480 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440483 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440486 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440489 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440491 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440494 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440496 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440499 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440502 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440504 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440506 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440509 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440511 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440514 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440516 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440519 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.444520 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440521 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440524 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440526 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440530 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440534 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440537 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440540 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440543 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440548 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440550 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440554 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440556 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440559 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440562 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440564 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440567 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440569 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440572 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440575 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440593 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.445040 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440596 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440599 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440602 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440604 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440607 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440610 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440612 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440615 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440617 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440620 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440622 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440625 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440628 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440631 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440634 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440636 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440640 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440643 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440645 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440648 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.445512 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440652 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440656 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440659 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440662 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440664 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440667 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440670 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440673 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440675 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440678 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440680 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440683 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440686 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440688 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440691 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440694 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440696 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440699 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440701 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440704 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.445992 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440706 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440709 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.440712 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440808 2584 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440824 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440834 2584 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440840 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440845 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440849 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440853 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440857 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440861 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440865 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440869 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440872 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440875 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440878 2584 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440881 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440884 2584 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440887 2584 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440890 2584 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440894 2584 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440897 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:30:52.446459 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440900 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440904 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440908 2584 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440912 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440915 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440918 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440921 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440925 2584 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440928 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440931 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440934 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440937 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440941 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440944 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440947 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440950 2584 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440953 2584 flags.go:64] FLAG: --enable-server="true" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440958 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440963 2584 flags.go:64] FLAG: --event-burst="100" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440966 2584 flags.go:64] FLAG: --event-qps="50" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440969 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440973 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440976 2584 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440981 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:30:52.447027 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440984 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440987 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440990 2584 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440993 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440996 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.440998 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441002 2584 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441005 2584 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441007 2584 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441010 2584 flags.go:64] FLAG: --feature-gates="" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441015 2584 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441018 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441021 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441024 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441027 2584 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441030 2584 flags.go:64] FLAG: --help="false" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441034 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441037 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441040 2584 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441043 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441046 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441049 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441052 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:30:52.447649 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441055 2584 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441058 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441063 2584 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441066 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441069 2584 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441072 2584 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441076 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441079 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441083 2584 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441085 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441088 2584 flags.go:64] FLAG: --lock-file="" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441091 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441094 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441097 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441103 2584 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441106 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441109 2584 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441112 2584 flags.go:64] FLAG: --logging-format="text" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441115 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441118 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441122 2584 flags.go:64] FLAG: --manifest-url="" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441124 2584 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441129 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441132 2584 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441136 2584 flags.go:64] FLAG: --max-pods="110" Nov 03 19:30:52.448210 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441139 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441142 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441145 2584 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441148 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441151 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441154 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441157 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441164 2584 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441167 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441170 2584 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441173 2584 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441176 2584 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441179 2584 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441183 2584 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441186 2584 flags.go:64] FLAG: --port="10250" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441188 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441191 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0bbb2aae223ccbeb8" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441194 2584 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441197 2584 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441201 2584 flags.go:64] FLAG: --register-node="true" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441203 2584 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441207 2584 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441210 2584 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441213 2584 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441216 2584 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441220 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:30:52.448817 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441223 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441226 2584 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441229 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441232 2584 flags.go:64] FLAG: --runonce="false" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441235 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441238 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441241 2584 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441243 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441246 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441249 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441252 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441255 2584 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441258 2584 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441260 2584 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441263 2584 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441266 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441269 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441272 2584 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441275 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441282 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441285 2584 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441288 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441291 2584 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441294 2584 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441297 2584 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:30:52.449419 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441300 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441303 2584 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441306 2584 flags.go:64] FLAG: --v="2" Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441311 2584 flags.go:64] FLAG: --version="false" Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441315 2584 flags.go:64] FLAG: --vmodule="" Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441318 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441322 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441452 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441455 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441458 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441462 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441465 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441467 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441470 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441472 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441475 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441478 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441480 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441483 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441485 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441487 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.450013 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441490 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441493 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441495 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441498 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441500 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441503 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441506 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441510 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441514 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441517 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441519 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441522 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441525 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441527 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441530 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441532 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441535 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441537 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441540 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.450505 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441542 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441545 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441547 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441550 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441553 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441555 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441558 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441560 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441563 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441565 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441568 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441570 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441572 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441575 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441590 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441593 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441596 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441598 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441602 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441605 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.450965 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441607 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441610 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441613 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441615 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441618 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441620 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441623 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441625 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441627 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441630 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441632 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441635 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441637 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441640 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441642 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441645 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441649 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441652 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441654 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.451432 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441657 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441660 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441662 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441667 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441669 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441672 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441674 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441677 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441679 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441682 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441685 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441687 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441690 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441692 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441695 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441697 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441700 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441702 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441705 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.451924 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441708 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441710 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441713 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441715 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441717 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441720 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441723 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441725 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441729 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441733 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441737 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441740 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441751 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441754 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441756 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.452358 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.441784 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441830 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441838 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441842 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441845 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441848 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441851 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441853 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441856 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441859 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441862 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441864 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441867 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441869 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441872 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441874 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441877 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441879 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441882 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441885 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.452773 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441887 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441890 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441892 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441895 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441897 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441900 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441902 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441906 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441909 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441912 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441915 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441917 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441920 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441923 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441926 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441929 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441932 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441934 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441937 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441939 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.453443 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441942 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441945 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441947 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441950 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441952 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441955 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441958 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441960 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441963 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441965 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441968 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441970 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441972 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441975 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441978 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441980 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441983 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441985 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441988 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.453955 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441992 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441994 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.441997 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442000 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442003 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442005 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442008 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442012 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442014 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442017 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442019 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442022 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442024 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442027 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442029 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442032 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442035 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442039 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442042 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.454390 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442044 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442048 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442052 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442056 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442059 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442062 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442065 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442068 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442071 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442073 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442076 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442078 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442081 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442085 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442088 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442090 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442093 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442096 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442098 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442101 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.454847 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442105 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442108 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442110 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442113 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442115 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442119 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442122 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442124 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.442127 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.448307 2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.448321 2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.449738 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.449745 2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449799 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449804 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449807 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449811 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449814 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449816 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449819 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.455323 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449822 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449825 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449827 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449830 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449833 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449835 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449838 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449840 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449843 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449848 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449853 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449857 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449860 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449863 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449866 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449869 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449871 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449874 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449876 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.455820 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449879 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449882 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449884 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449888 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449890 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449893 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449896 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449898 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449901 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449904 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449906 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449909 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449911 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449914 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449916 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449919 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449921 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449924 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449926 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449928 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.456271 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449931 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449934 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449936 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449940 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449944 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449948 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449950 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449953 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449956 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449958 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449960 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449963 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449965 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449968 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449970 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449973 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449976 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449979 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449981 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.456753 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449983 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449986 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449989 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449991 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449994 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449997 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.449999 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450002 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450004 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450007 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450009 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450012 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450014 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450017 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450019 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450022 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450025 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450028 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450031 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450033 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.457232 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450036 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450038 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450041 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450043 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450046 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450048 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450051 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450053 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450056 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450059 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450061 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450063 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450066 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450069 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450071 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450074 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450077 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450079 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450082 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450084 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.457716 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450087 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.450114 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450266 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450271 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450274 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450277 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450279 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450282 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450285 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450287 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450290 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450293 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450296 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450298 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450301 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450303 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.458177 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450305 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450308 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450311 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450313 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450316 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450319 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450321 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450324 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450326 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450329 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450331 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450334 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450336 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450339 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450341 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450344 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450346 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450349 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450351 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450354 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.458550 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450356 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450359 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450361 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450364 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450366 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450369 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450371 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450376 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450380 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450383 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450386 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450389 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450391 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450394 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450396 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450399 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450401 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450405 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450407 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.459088 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450409 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450412 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450414 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450417 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450419 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450422 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450424 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450427 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450429 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450432 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450434 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450437 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450439 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450443 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450447 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450449 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450452 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450455 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450457 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.459527 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450460 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450462 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450465 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450468 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450470 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450473 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450475 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450478 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450480 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450483 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450485 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450488 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450491 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450493 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450496 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450498 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450500 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450503 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450505 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450508 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.459983 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450511 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450513 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450516 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450519 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450521 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450523 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450526 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450528 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450531 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450533 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450536 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450538 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450540 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450543 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.450565 2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450614 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:30:52.460454 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450619 2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450622 2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450625 2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450628 2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450631 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450634 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450637 2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450639 2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450642 2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450644 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450647 2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450649 2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450652 2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450656 2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450660 2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450664 2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450667 2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450670 2584 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450673 2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:30:52.460840 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450675 2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450678 2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450680 2584 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450682 2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450685 2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450688 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450690 2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450692 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450695 2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450697 2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450700 2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450702 2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450705 2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450708 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450710 2584 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450713 2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450715 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450718 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450721 2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450723 2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:30:52.461277 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450726 2584 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450728 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450731 2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450734 2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450736 2584 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450739 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450741 2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450744 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450746 2584 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450749 2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450751 2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450754 2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450756 2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450759 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450761 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450763 2584 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450766 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450769 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450771 2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450774 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:30:52.461846 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450776 2584 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450779 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450781 2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450784 2584 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450795 2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450798 2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450801 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450803 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450806 2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450808 2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450811 2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450814 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450816 2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450819 2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450821 2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450824 2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450826 2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450829 2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450831 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:30:52.462308 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450834 2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450837 2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450839 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450842 2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450844 2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450847 2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450849 2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450852 2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450855 2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450857 2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450863 2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450865 2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450868 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450870 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450873 2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450876 2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450880 2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450883 2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450886 2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:30:52.462759 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450890 2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450892 2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450895 2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450898 2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450900 2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450903 2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450905 2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:52.450908 2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.450954 2584 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.456881 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.459967 2584 server.go:1014] "Starting client certificate rotation" Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.460063 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.460094 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:30:52.463198 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.462513 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:30:52.463544 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.463527 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:30:52.474659 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.474642 2584 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:30:52.478522 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.478506 2584 log.go:25] "Validated CRI v1 image API" Nov 03 19:30:52.479339 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.479315 2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:30:52.483948 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.483930 2584 fs.go:135] Filesystem UUIDs: map[0909f9ba-24bc-4eca-8b36-eb44a5b68d76:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d5567e3c-870c-4519-8921-8cf21cb58727:/dev/nvme0n1p3] Nov 03 19:30:52.484007 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.483947 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 19:30:52.489498 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.489392 2584 manager.go:217] Machine: {Timestamp:2026-11-03 19:30:52.487974868 +0000 UTC m=+0.352977306 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098884 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e69c77088d9370497d5b1f162206 SystemUUID:ec29e69c-7708-8d93-7049-7d5b1f162206 BootID:c19d015b-d39c-4777-ab56-b9c31eb6a31a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fe:a5:b6:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fe:a5:b6:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:1d:dc:62:a4:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:30:52.489622 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.489499 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:30:52.489622 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.489614 2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:30:52.492605 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.492562 2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:30:52.492782 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.492608 2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-204.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:30:52.493385 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.493372 2584 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:30:52.493414 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.493388 2584 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:30:52.493441 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.493417 2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:30:52.493441 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.493435 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:30:52.494234 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.494223 2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:30:52.498393 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498369 2584 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:30:52.498547 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498534 2584 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:30:52.498617 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498576 2584 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:30:52.498617 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498612 2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:30:52.498717 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498632 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:30:52.498717 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498659 2584 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:30:52.498717 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498671 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:30:52.498888 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.498824 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:30:52.499120 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.499103 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:30:52.503203 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.503183 2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:30:52.510385 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.510370 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:30:52.511203 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511190 2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:30:52.511256 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511222 2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:30:52.511394 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511384 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:30:52.511429 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511400 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:30:52.511429 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511406 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:30:52.511429 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511412 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:30:52.511429 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511418 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:30:52.511429 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511423 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:30:52.512000 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.511989 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:30:52.512000 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.512001 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:30:52.512069 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.512009 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:30:52.512069 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.512015 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:30:52.512069 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.512024 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:30:52.512069 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.512034 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:30:52.512688 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.512679 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:30:52.512725 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.512689 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:30:52.516472 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.516458 2584 server.go:1263] "Started kubelet" Nov 03 19:30:52.516610 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.516547 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:30:52.516675 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.516602 2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:30:52.516675 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.516632 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:30:52.517167 ip-10-0-133-204 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:30:52.519478 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.519416 2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:30:52.520798 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.520785 2584 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:30:52.521529 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.521369 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 19:30:52.521529 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.521507 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-204.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 19:30:52.521644 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.521526 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-204.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 19:30:52.522441 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.521344 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-204.us-east-2.compute.internal.18e49fcc8eb15a2a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-204.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-204.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:30:52.516432426 +0000 UTC m=+0.381434864,LastTimestamp:2026-11-03 19:30:52.516432426 +0000 UTC m=+0.381434864,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-204.us-east-2.compute.internal,}" Nov 03 19:30:52.523645 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.523623 2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 19:30:52.524227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.524211 2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:30:52.524227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.524224 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:30:52.524350 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.524284 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:52.524957 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.524919 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:30:52.525080 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525052 2584 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:30:52.525080 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525074 2584 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:30:52.525241 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525200 2584 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:30:52.525241 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525208 2584 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:30:52.525663 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525646 2584 factory.go:55] Registering systemd factory Nov 03 19:30:52.525663 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525666 2584 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:30:52.525957 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525942 2584 factory.go:153] Registering CRI-O factory Nov 03 19:30:52.526009 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.525961 2584 factory.go:223] Registration of the crio container factory successfully Nov 03 19:30:52.526009 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.526006 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:30:52.526101 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.526021 2584 factory.go:103] Registering Raw factory Nov 03 19:30:52.526101 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.526037 2584 manager.go:1199] Started watching for new ooms in manager Nov 03 19:30:52.527279 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.527262 2584 manager.go:319] Starting recovery of all containers Nov 03 19:30:52.528325 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.528304 2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-204.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 19:30:52.537986 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.537966 2584 manager.go:324] Recovery completed Nov 03 19:30:52.542063 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.542042 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:52.543491 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.543473 2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 19:30:52.544216 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.544199 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:52.544274 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.544229 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:52.544274 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.544244 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:52.544768 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.544756 2584 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:30:52.544768 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.544767 2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:30:52.544856 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.544777 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:52.544856 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.544781 2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:30:52.545562 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.545549 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:52.545680 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.545577 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:52.545738 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.545694 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:52.547099 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.547085 2584 policy_none.go:50] "Start" Nov 03 19:30:52.547099 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.547101 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:30:52.547207 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.547111 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:30:52.549053 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.549039 2584 policy_none.go:44] "Start" Nov 03 19:30:52.550266 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.550251 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r8b6f" Nov 03 19:30:52.553398 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.553337 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-204.us-east-2.compute.internal.18e49fcc905942f8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-204.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-204.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-204.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:30:52.544213752 +0000 UTC m=+0.409216189,LastTimestamp:2026-11-03 19:30:52.544213752 +0000 UTC m=+0.409216189,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-204.us-east-2.compute.internal,}" Nov 03 19:30:52.555837 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.555821 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r8b6f" Nov 03 19:30:52.590306 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.590272 2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:30:52.593360 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.593345 2584 manager.go:341] "Starting Device Plugin manager" Nov 03 19:30:52.593837 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.593819 2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:30:52.593892 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.593839 2584 server.go:87] "Starting device plugin registration server" Nov 03 19:30:52.594072 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.594061 2584 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:30:52.594101 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.594076 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:30:52.594177 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.594164 2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:30:52.594274 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.594229 2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:30:52.594274 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.594237 2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:30:52.594815 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.594794 2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:30:52.594891 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.594841 2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:52.696371 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.694546 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:52.696371 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.695362 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:52.696371 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.695387 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:52.696371 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.695397 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:52.696371 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.695421 2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.703884 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.703869 2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.703954 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.703892 2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-204.us-east-2.compute.internal\": node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:52.720747 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.720730 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:52.741359 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.741335 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:30:52.742409 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.742392 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:30:52.742486 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.742415 2584 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:30:52.742486 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.742436 2584 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:30:52.742486 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.742473 2584 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 19:30:52.753114 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.753096 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:30:52.820978 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.820952 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:52.843374 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.843353 2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal"] Nov 03 19:30:52.843429 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.843411 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:52.844323 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.844307 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:52.844395 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.844339 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:52.844395 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.844351 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:52.846312 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.846301 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:52.846466 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.846452 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.846501 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.846481 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:52.847036 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.847021 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:52.847147 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.847041 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:52.847147 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.847048 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:52.847147 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.847058 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:52.847147 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.847060 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:52.847147 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.847070 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:52.849084 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.849069 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.849136 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.849095 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:52.849793 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.849779 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:52.849864 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.849804 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:52.849864 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.849816 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:52.875745 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.875730 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" node="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.877712 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.877697 2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" node="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.921384 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:52.921368 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:52.926284 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.926267 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.926375 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.926291 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:52.926375 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:52.926307 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70af56b747e8dbbb62b5b41a5fc7a4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"70af56b747e8dbbb62b5b41a5fc7a4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.022072 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.022015 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.027372 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.027354 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.027433 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.027381 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.027433 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.027398 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70af56b747e8dbbb62b5b41a5fc7a4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"70af56b747e8dbbb62b5b41a5fc7a4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.027433 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.027430 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70af56b747e8dbbb62b5b41a5fc7a4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"70af56b747e8dbbb62b5b41a5fc7a4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.027555 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.027455 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.027555 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.027461 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.122544 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.122524 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.179077 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.179058 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.181216 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.181199 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:53.222815 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.222786 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.323667 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.323465 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.426047 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.426024 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.457950 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.457934 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 19:30:53.458342 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.458077 2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:30:53.526889 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.526858 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.560786 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.560721 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:25:52 +0000 UTC" deadline="2028-05-01 21:28:26.75034991 +0000 UTC" Nov 03 19:30:53.560786 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.560752 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13081h57m33.189602331s" Nov 03 19:30:53.627405 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.627388 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.702352 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.702310 2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:30:53.703326 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.703309 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:30:53.703417 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.703348 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:30:53.703417 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.703365 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:30:53.727947 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.727928 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.829007 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.828918 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.929726 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:53.929703 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:53.932962 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:53.932925 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70af56b747e8dbbb62b5b41a5fc7a4ec.slice/crio-83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc WatchSource:0}: Error finding container 83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc: Status 404 returned error can't find the container with id 83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc Nov 03 19:30:53.933174 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:53.933156 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300bf0c34ec8efd4851a1973652985e3.slice/crio-7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9 WatchSource:0}: Error finding container 7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9: Status 404 returned error can't find the container with id 7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9 Nov 03 19:30:53.937814 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.937794 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:30:53.950630 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:53.950606 2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:30:54.000826 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.000800 2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:30:54.030423 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.030401 2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:30:54.054039 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.054022 2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:30:54.129985 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.129934 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:54.143744 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.143723 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:30:54.144744 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.144731 2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:30:54.152725 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.152710 2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:30:54.338934 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.338904 2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:30:54.500934 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.500860 2584 apiserver.go:52] "Watching apiserver" Nov 03 19:30:54.508691 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.508648 2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:30:54.509116 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.509041 2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ns5zn","openshift-image-registry/node-ca-zj9jh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal","openshift-multus/multus-k6fjx","openshift-multus/network-metrics-daemon-bbtrq","kube-system/konnectivity-agent-8cmvr","openshift-cluster-node-tuning-operator/tuned-vhgfr","openshift-multus/multus-additional-cni-plugins-jf9qv","openshift-network-diagnostics/network-check-target-pp285","openshift-network-operator/iptables-alerter-629fw","openshift-ovn-kubernetes/ovnkube-node-xss4s","kube-system/global-pull-secret-syncer-8z8g9","kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5"] Nov 03 19:30:54.511660 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.511639 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.514681 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.514652 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.515841 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.515748 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:30:54.515841 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.515764 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:30:54.515841 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.515818 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6p8wh\"" Nov 03 19:30:54.518045 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.518027 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.520430 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.520411 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.523483 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.523436 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:54.523581 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.523536 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:30:54.524566 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.524547 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 19:30:54.527241 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.526807 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.529314 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.529298 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:54.529447 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.529363 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:30:54.531782 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.531763 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.536175 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536155 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.536291 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536212 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-tmp\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.536291 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536247 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-system-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.536291 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536278 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-cni-binary-copy\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.536409 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536302 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-conf-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.536409 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536328 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-slash\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.536409 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536352 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-ovn\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.536409 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536374 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-env-overrides\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.537076 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536397 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-kubelet\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.537076 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.536974 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-sys\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.537076 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537014 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-host\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.537076 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537049 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-cnibin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.537076 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537084 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-modprobe-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537113 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-netns\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537151 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537184 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537219 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-var-lib-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537248 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537281 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-daemon-config\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537313 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-bin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537348 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d56c5c2-1d76-490b-b853-3e8d15662fa5-konnectivity-ca\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537388 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537421 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-run\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.537471 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.537448 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-var-lib-kubelet\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.538780 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.538717 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-kubelet\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.538898 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.538848 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.538959 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.538935 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-netd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.539034 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.538973 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.539034 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539025 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.539129 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539056 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-lib-modules\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.539129 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539088 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-systemd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.539225 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539139 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-tuned\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.539225 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539169 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqnq\" (UniqueName: \"kubernetes.io/projected/e17fbd17-d288-4823-bad6-cdd6d37e5c15-kube-api-access-lfqnq\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.539225 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539201 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.539369 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539227 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-socket-dir-parent\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.539369 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539260 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-kubernetes\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.539369 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539292 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-script-lib\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539605 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9vw\" (UniqueName: \"kubernetes.io/projected/c8b6db40-add7-4eca-9480-e174e021fdc7-kube-api-access-vf9vw\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539691 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-netns\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539740 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtc5z\" (UniqueName: \"kubernetes.io/projected/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kube-api-access-wtc5z\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539786 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539849 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-bin\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539906 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-config\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.539950 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540030 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-multus\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540123 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6w68\" (UniqueName: \"kubernetes.io/projected/d7f898aa-1924-459a-af1c-2242ac527e2a-kube-api-access-q6w68\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540192 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-conf\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540258 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-systemd-units\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540411 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-hostroot\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540459 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysconfig\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540507 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b6db40-add7-4eca-9480-e174e021fdc7-ovn-node-metrics-cert\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540538 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-device-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.542690 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540569 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-etc-kubernetes\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540602 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d56c5c2-1d76-490b-b853-3e8d15662fa5-agent-certs\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540627 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-etc-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540675 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-node-log\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540705 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-log-socket\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540735 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8zx\" (UniqueName: \"kubernetes.io/projected/90d8527c-7221-45b1-b2c5-9397fa8db4e3-kube-api-access-8k8zx\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540766 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-systemd\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540793 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540862 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-k8s-cni-cncf-io\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.540948 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-multus-certs\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.541924 2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.542032 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.543243 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.542736 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.545681 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.545661 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.545766 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.545740 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:30:54.552530 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.552505 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ztm4z" Nov 03 19:30:54.561120 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.561098 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ztm4z" Nov 03 19:30:54.625547 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.625082 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:30:54.625547 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.625121 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:30:54.625547 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.625174 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:30:54.626001 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.625977 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:30:54.626001 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.625997 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:30:54.626158 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626131 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:30:54.626215 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626157 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:30:54.626316 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626274 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:30:54.626378 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626366 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:30:54.626440 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626423 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pq7nf\"" Nov 03 19:30:54.626494 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626459 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:30:54.626550 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626517 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:30:54.626606 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626549 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:30:54.626606 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626574 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:30:54.626741 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626603 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9h8df\"" Nov 03 19:30:54.626741 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626727 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:30:54.626858 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626790 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wllhj\"" Nov 03 19:30:54.626858 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626825 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:30:54.626858 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626855 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:30:54.627046 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626912 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:30:54.627046 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.626462 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-glk7q\"" Nov 03 19:30:54.627157 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.627066 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kqxn2\"" Nov 03 19:30:54.627193 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.627166 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:30:54.627250 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.627236 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:30:54.627661 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.627638 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgldx\"" Nov 03 19:30:54.627764 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.627722 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:30:54.628187 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.628173 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:30:54.628275 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.628223 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bh6bb\"" Nov 03 19:30:54.628334 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.628283 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:30:54.628393 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.628364 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:30:54.628451 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.628435 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:30:54.628504 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.628488 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:30:54.629508 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.629467 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6x7r\"" Nov 03 19:30:54.629629 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.629514 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:30:54.641973 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.641948 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-config\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.642072 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.641999 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.642072 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642029 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-multus\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.642072 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642053 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6w68\" (UniqueName: \"kubernetes.io/projected/d7f898aa-1924-459a-af1c-2242ac527e2a-kube-api-access-q6w68\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.642235 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642082 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6qm\" (UniqueName: \"kubernetes.io/projected/0225e321-66c4-4269-81d4-051793673f4d-kube-api-access-fw6qm\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.642235 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642112 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.642235 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642138 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5q4r\" (UniqueName: \"kubernetes.io/projected/8bf911cd-cd3c-4990-81ce-80d9ef41f214-kube-api-access-t5q4r\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.642235 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642176 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-conf\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.642235 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642199 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-systemd-units\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.642235 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642222 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-hostroot\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642247 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0225e321-66c4-4269-81d4-051793673f4d-hosts-file\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642277 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysconfig\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642302 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b6db40-add7-4eca-9480-e174e021fdc7-ovn-node-metrics-cert\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642335 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-device-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642360 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-etc-kubernetes\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642384 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d56c5c2-1d76-490b-b853-3e8d15662fa5-agent-certs\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642409 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-etc-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642434 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-node-log\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642459 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-log-socket\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642484 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8zx\" (UniqueName: \"kubernetes.io/projected/90d8527c-7221-45b1-b2c5-9397fa8db4e3-kube-api-access-8k8zx\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:54.642525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642510 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cnibin\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642535 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-systemd\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642562 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642587 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-k8s-cni-cncf-io\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642612 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-multus-certs\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642652 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642678 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fbf00d3-e30f-4399-b770-6545f69f15ce-host\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642703 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.642727 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-tmp\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.643088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643056 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-system-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.643577 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643091 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-cni-binary-copy\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.643577 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643099 2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:30:54.643577 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643144 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-conf-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.643577 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643155 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-etc-kubernetes\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.643577 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643170 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.643837 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643639 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-conf-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.643837 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643762 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.643837 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.643832 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-multus\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.644402 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.644380 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-conf\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.644468 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.644438 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-systemd-units\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.644525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.644475 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-hostroot\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.644569 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.644532 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysconfig\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.645191 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645167 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-device-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.645710 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645679 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-cni-binary-copy\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.645793 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645749 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zkv\" (UniqueName: \"kubernetes.io/projected/4e4278cb-b8ca-421a-804b-dc7c728ba346-kube-api-access-d2zkv\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.645852 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645790 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-slash\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.645852 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645825 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-ovn\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.645952 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645896 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-env-overrides\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.645952 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645923 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-kubelet\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.646062 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645954 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-sys\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.646062 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.645985 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-host\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.646062 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646041 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-cnibin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.646207 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646076 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-system-cni-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.646207 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646168 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-modprobe-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.646207 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646201 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-netns\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.646360 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646228 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.646360 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646259 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.646360 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646293 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-var-lib-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.646360 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646326 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.646360 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646357 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-daemon-config\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.646604 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646387 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb56k\" (UniqueName: \"kubernetes.io/projected/3fbf00d3-e30f-4399-b770-6545f69f15ce-kube-api-access-nb56k\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.646604 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646424 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-kubelet-config\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.646604 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646457 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-bin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.646604 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646521 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-ovn\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.646828 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646621 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-system-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.646828 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.646684 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-slash\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.647065 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647045 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-env-overrides\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.647138 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647107 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-kubelet\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.647194 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647156 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-sys\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.647256 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647201 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-host\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.647309 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647258 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-cnibin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.647368 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647342 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-modprobe-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.647442 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647395 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-netns\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.647484 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647456 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.647598 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647527 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.647598 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647585 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-var-lib-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.647769 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.647708 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.648177 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.648159 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-daemon-config\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.648327 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.648266 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-bin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.648386 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.648322 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d56c5c2-1d76-490b-b853-3e8d15662fa5-konnectivity-ca\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.648442 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.648428 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.648497 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.648455 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-run\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.649799 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.649880 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-var-lib-kubelet\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.649939 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-kubelet\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.649975 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650009 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-netd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650060 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650094 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650134 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e4278cb-b8ca-421a-804b-dc7c728ba346-iptables-alerter-script\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650169 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e4278cb-b8ca-421a-804b-dc7c728ba346-host-slash\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-lib-modules\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650232 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-systemd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650282 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-tuned\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650283 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d56c5c2-1d76-490b-b853-3e8d15662fa5-konnectivity-ca\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqnq\" (UniqueName: \"kubernetes.io/projected/e17fbd17-d288-4823-bad6-cdd6d37e5c15-kube-api-access-lfqnq\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650432 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-run\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650442 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.650713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650472 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-socket-dir-parent\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650506 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-kubernetes\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650509 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-systemd\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650541 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-script-lib\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650578 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9vw\" (UniqueName: \"kubernetes.io/projected/c8b6db40-add7-4eca-9480-e174e021fdc7-kube-api-access-vf9vw\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650628 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-netns\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650698 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0225e321-66c4-4269-81d4-051793673f4d-tmp-dir\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650734 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650783 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtc5z\" (UniqueName: \"kubernetes.io/projected/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kube-api-access-wtc5z\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650819 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650857 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3fbf00d3-e30f-4399-b770-6545f69f15ce-serviceca\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650890 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.650973 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-dbus\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651008 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-bin\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651127 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-bin\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651172 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-var-lib-kubelet\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651237 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-log-socket\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.651476 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651250 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.652295 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651271 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-k8s-cni-cncf-io\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.652295 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651288 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-multus-certs\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.652295 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651370 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-node-log\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.652295 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.644802 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-config\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.652295 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.651997 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-systemd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.652526 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.652387 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-kubelet\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.652526 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.652465 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.652526 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.652500 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-socket-dir-parent\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.652650 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.652552 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-kubernetes\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.652695 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.652675 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-netns\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.652878 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.652859 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:54.652951 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.652940 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs podName:90d8527c-7221-45b1-b2c5-9397fa8db4e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:55.152913058 +0000 UTC m=+3.017915503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs") pod "network-metrics-daemon-bbtrq" (UID: "90d8527c-7221-45b1-b2c5-9397fa8db4e3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:54.653215 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.653193 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-netd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.653808 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.653286 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.653808 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.653542 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.653808 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.653569 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-tmp\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.653808 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.653620 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-script-lib\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.653808 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.653652 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.653808 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.653768 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-etc-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.654230 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.654026 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-lib-modules\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.654230 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.654157 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d56c5c2-1d76-490b-b853-3e8d15662fa5-agent-certs\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.655890 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.655857 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b6db40-add7-4eca-9480-e174e021fdc7-ovn-node-metrics-cert\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.657593 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.657548 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-tuned\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.664067 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.664047 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6w68\" (UniqueName: \"kubernetes.io/projected/d7f898aa-1924-459a-af1c-2242ac527e2a-kube-api-access-q6w68\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.672819 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.672794 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8zx\" (UniqueName: \"kubernetes.io/projected/90d8527c-7221-45b1-b2c5-9397fa8db4e3-kube-api-access-8k8zx\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:54.675205 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.674864 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9vw\" (UniqueName: \"kubernetes.io/projected/c8b6db40-add7-4eca-9480-e174e021fdc7-kube-api-access-vf9vw\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.675534 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.675513 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqnq\" (UniqueName: \"kubernetes.io/projected/e17fbd17-d288-4823-bad6-cdd6d37e5c15-kube-api-access-lfqnq\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.676382 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.676360 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtc5z\" (UniqueName: \"kubernetes.io/projected/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kube-api-access-wtc5z\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.748078 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.747999 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"70af56b747e8dbbb62b5b41a5fc7a4ec","Type":"ContainerStarted","Data":"83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc"} Nov 03 19:30:54.749428 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.749403 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"300bf0c34ec8efd4851a1973652985e3","Type":"ContainerStarted","Data":"7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9"} Nov 03 19:30:54.751991 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.751923 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6qm\" (UniqueName: \"kubernetes.io/projected/0225e321-66c4-4269-81d4-051793673f4d-kube-api-access-fw6qm\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.752094 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.751993 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.752094 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752022 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5q4r\" (UniqueName: \"kubernetes.io/projected/8bf911cd-cd3c-4990-81ce-80d9ef41f214-kube-api-access-t5q4r\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.752094 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752050 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0225e321-66c4-4269-81d4-051793673f4d-hosts-file\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.752253 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752082 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cnibin\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.752253 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752149 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:54.752253 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752174 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fbf00d3-e30f-4399-b770-6545f69f15ce-host\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.752253 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752205 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.752253 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752235 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752262 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zkv\" (UniqueName: \"kubernetes.io/projected/4e4278cb-b8ca-421a-804b-dc7c728ba346-kube-api-access-d2zkv\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752298 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-system-cni-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752331 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb56k\" (UniqueName: \"kubernetes.io/projected/3fbf00d3-e30f-4399-b770-6545f69f15ce-kube-api-access-nb56k\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752364 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-kubelet-config\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752402 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e4278cb-b8ca-421a-804b-dc7c728ba346-iptables-alerter-script\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752427 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e4278cb-b8ca-421a-804b-dc7c728ba346-host-slash\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752461 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0225e321-66c4-4269-81d4-051793673f4d-tmp-dir\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.752507 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752489 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.752930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752530 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3fbf00d3-e30f-4399-b770-6545f69f15ce-serviceca\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.752930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752556 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.752930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752581 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-dbus\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.752930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.752706 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-dbus\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.753203 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-system-cni-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.753268 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fbf00d3-e30f-4399-b770-6545f69f15ce-host\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.753362 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.753430 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret podName:7bac36d0-ca54-4a9c-83cd-c58ba9f58c37 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:55.253412252 +0000 UTC m=+3.118414677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret") pod "global-pull-secret-syncer-8z8g9" (UID: "7bac36d0-ca54-4a9c-83cd-c58ba9f58c37") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.753628 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0225e321-66c4-4269-81d4-051793673f4d-hosts-file\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.753648 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.753684 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.753696 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cnibin\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.753961 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0225e321-66c4-4269-81d4-051793673f4d-tmp-dir\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.754227 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.754023 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-kubelet-config\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:54.754757 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.754311 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e4278cb-b8ca-421a-804b-dc7c728ba346-host-slash\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.755009 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.754942 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.755320 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.755298 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e4278cb-b8ca-421a-804b-dc7c728ba346-iptables-alerter-script\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.755443 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.755371 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3fbf00d3-e30f-4399-b770-6545f69f15ce-serviceca\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.755806 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.755781 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.759466 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.759227 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:30:54.759466 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.759272 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:30:54.759466 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.759286 2584 projected.go:196] Error preparing data for projected volume kube-api-access-x74s2 for pod openshift-network-diagnostics/network-check-target-pp285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:54.759466 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:54.759383 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2 podName:656bc97e-5a5b-4334-8369-bc842712cf02 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:55.259365411 +0000 UTC m=+3.124367856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x74s2" (UniqueName: "kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2") pod "network-check-target-pp285" (UID: "656bc97e-5a5b-4334-8369-bc842712cf02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:54.761904 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.761885 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb56k\" (UniqueName: \"kubernetes.io/projected/3fbf00d3-e30f-4399-b770-6545f69f15ce-kube-api-access-nb56k\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.762109 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.762083 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6qm\" (UniqueName: \"kubernetes.io/projected/0225e321-66c4-4269-81d4-051793673f4d-kube-api-access-fw6qm\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.762709 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.762691 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zkv\" (UniqueName: \"kubernetes.io/projected/4e4278cb-b8ca-421a-804b-dc7c728ba346-kube-api-access-d2zkv\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.763259 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.763227 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5q4r\" (UniqueName: \"kubernetes.io/projected/8bf911cd-cd3c-4990-81ce-80d9ef41f214-kube-api-access-t5q4r\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.828366 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.828337 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:30:54.841783 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.841750 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17fbd17_d288_4823_bad6_cdd6d37e5c15.slice/crio-58c32fd8caf54556cbfa4d03f48f7fa6d0181bb6842a96fdbcce299d1589d5f6 WatchSource:0}: Error finding container 58c32fd8caf54556cbfa4d03f48f7fa6d0181bb6842a96fdbcce299d1589d5f6: Status 404 returned error can't find the container with id 58c32fd8caf54556cbfa4d03f48f7fa6d0181bb6842a96fdbcce299d1589d5f6 Nov 03 19:30:54.855090 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.855070 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:30:54.858086 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.858056 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:30:54.867434 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.867399 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b6db40_add7_4eca_9480_e174e021fdc7.slice/crio-4cbe6aa8bcb1b8a94796ba0f2d28fc89e35cbd8e5f0bcc75d46dde2430abc478 WatchSource:0}: Error finding container 4cbe6aa8bcb1b8a94796ba0f2d28fc89e35cbd8e5f0bcc75d46dde2430abc478: Status 404 returned error can't find the container with id 4cbe6aa8bcb1b8a94796ba0f2d28fc89e35cbd8e5f0bcc75d46dde2430abc478 Nov 03 19:30:54.869188 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.869167 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:30:54.877316 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.877297 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6fjx" Nov 03 19:30:54.881266 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.881240 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9309b9f0_2a67_4c14_8af9_8759348ae5d5.slice/crio-ebf57f6c0b543b94e8a7226744d3ac9a460f414a560a799b9f092954c3975b05 WatchSource:0}: Error finding container ebf57f6c0b543b94e8a7226744d3ac9a460f414a560a799b9f092954c3975b05: Status 404 returned error can't find the container with id ebf57f6c0b543b94e8a7226744d3ac9a460f414a560a799b9f092954c3975b05 Nov 03 19:30:54.887325 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.887301 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f898aa_1924_459a_af1c_2242ac527e2a.slice/crio-d3b8a09106b087bfa2f77220293563f980f76ac6e654187168cc595690bb42a5 WatchSource:0}: Error finding container d3b8a09106b087bfa2f77220293563f980f76ac6e654187168cc595690bb42a5: Status 404 returned error can't find the container with id d3b8a09106b087bfa2f77220293563f980f76ac6e654187168cc595690bb42a5 Nov 03 19:30:54.900284 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.900264 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:30:54.902595 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.902556 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:30:54.905196 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.905154 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:30:54.910402 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:54.910381 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:30:54.915893 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.915865 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf911cd_cd3c_4990_81ce_80d9ef41f214.slice/crio-83c3e661343f8afd2c02edc813790851a08b88c0f168ba1f7b251012ed1bdbc8 WatchSource:0}: Error finding container 83c3e661343f8afd2c02edc813790851a08b88c0f168ba1f7b251012ed1bdbc8: Status 404 returned error can't find the container with id 83c3e661343f8afd2c02edc813790851a08b88c0f168ba1f7b251012ed1bdbc8 Nov 03 19:30:54.920150 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.917694 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbf00d3_e30f_4399_b770_6545f69f15ce.slice/crio-957407ed50e65883014afa775cc085e9500e73441a343a28f2228df26942aaac WatchSource:0}: Error finding container 957407ed50e65883014afa775cc085e9500e73441a343a28f2228df26942aaac: Status 404 returned error can't find the container with id 957407ed50e65883014afa775cc085e9500e73441a343a28f2228df26942aaac Nov 03 19:30:54.922808 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.922784 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4278cb_b8ca_421a_804b_dc7c728ba346.slice/crio-9aabebd259389a1a32ec28a8ebe43dd4ee6f51d0a1c4dd24de6199f9e7dabef7 WatchSource:0}: Error finding container 9aabebd259389a1a32ec28a8ebe43dd4ee6f51d0a1c4dd24de6199f9e7dabef7: Status 404 returned error can't find the container with id 9aabebd259389a1a32ec28a8ebe43dd4ee6f51d0a1c4dd24de6199f9e7dabef7 Nov 03 19:30:54.927267 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:30:54.927244 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0225e321_66c4_4269_81d4_051793673f4d.slice/crio-a69fa057d93699cf33a3f8219fd1adb86b4d4c02f763e9532e1f4c6137ae1e2e WatchSource:0}: Error finding container a69fa057d93699cf33a3f8219fd1adb86b4d4c02f763e9532e1f4c6137ae1e2e: Status 404 returned error can't find the container with id a69fa057d93699cf33a3f8219fd1adb86b4d4c02f763e9532e1f4c6137ae1e2e Nov 03 19:30:55.156713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.156625 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:55.156868 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.156794 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:55.156941 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.156870 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs podName:90d8527c-7221-45b1-b2c5-9397fa8db4e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:56.156847482 +0000 UTC m=+4.021849955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs") pod "network-metrics-daemon-bbtrq" (UID: "90d8527c-7221-45b1-b2c5-9397fa8db4e3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:55.257927 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.257887 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:55.258122 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.258059 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:55.258122 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.258121 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret podName:7bac36d0-ca54-4a9c-83cd-c58ba9f58c37 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:56.258107076 +0000 UTC m=+4.123109506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret") pod "global-pull-secret-syncer-8z8g9" (UID: "7bac36d0-ca54-4a9c-83cd-c58ba9f58c37") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:55.358686 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.358647 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:55.358859 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.358824 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:30:55.358859 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.358843 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:30:55.358859 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.358857 2584 projected.go:196] Error preparing data for projected volume kube-api-access-x74s2 for pod openshift-network-diagnostics/network-check-target-pp285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:55.359128 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.358913 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2 podName:656bc97e-5a5b-4334-8369-bc842712cf02 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:56.358895725 +0000 UTC m=+4.223898159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x74s2" (UniqueName: "kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2") pod "network-check-target-pp285" (UID: "656bc97e-5a5b-4334-8369-bc842712cf02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:55.564380 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.564056 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:54 +0000 UTC" deadline="2028-05-30 05:34:52.604549843 +0000 UTC" Nov 03 19:30:55.564380 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.564096 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13762h3m57.040457772s" Nov 03 19:30:55.743795 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.743204 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:55.743795 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:55.743340 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:30:55.765882 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.765743 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"4cbe6aa8bcb1b8a94796ba0f2d28fc89e35cbd8e5f0bcc75d46dde2430abc478"} Nov 03 19:30:55.773790 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.773709 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ns5zn" event={"ID":"0225e321-66c4-4269-81d4-051793673f4d","Type":"ContainerStarted","Data":"a69fa057d93699cf33a3f8219fd1adb86b4d4c02f763e9532e1f4c6137ae1e2e"} Nov 03 19:30:55.781460 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.781427 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-629fw" event={"ID":"4e4278cb-b8ca-421a-804b-dc7c728ba346","Type":"ContainerStarted","Data":"9aabebd259389a1a32ec28a8ebe43dd4ee6f51d0a1c4dd24de6199f9e7dabef7"} Nov 03 19:30:55.791987 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.791907 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" event={"ID":"9309b9f0-2a67-4c14-8af9-8759348ae5d5","Type":"ContainerStarted","Data":"ebf57f6c0b543b94e8a7226744d3ac9a460f414a560a799b9f092954c3975b05"} Nov 03 19:30:55.798962 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.798933 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8cmvr" event={"ID":"2d56c5c2-1d76-490b-b853-3e8d15662fa5","Type":"ContainerStarted","Data":"b9e625b9dd71b93087b9217529781377464a33122fe62cd15c87d02677fdebd9"} Nov 03 19:30:55.812206 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.812134 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" event={"ID":"e17fbd17-d288-4823-bad6-cdd6d37e5c15","Type":"ContainerStarted","Data":"58c32fd8caf54556cbfa4d03f48f7fa6d0181bb6842a96fdbcce299d1589d5f6"} Nov 03 19:30:55.826217 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.826096 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerStarted","Data":"83c3e661343f8afd2c02edc813790851a08b88c0f168ba1f7b251012ed1bdbc8"} Nov 03 19:30:55.833002 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.832927 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zj9jh" event={"ID":"3fbf00d3-e30f-4399-b770-6545f69f15ce","Type":"ContainerStarted","Data":"957407ed50e65883014afa775cc085e9500e73441a343a28f2228df26942aaac"} Nov 03 19:30:55.838675 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:55.838635 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6fjx" event={"ID":"d7f898aa-1924-459a-af1c-2242ac527e2a","Type":"ContainerStarted","Data":"d3b8a09106b087bfa2f77220293563f980f76ac6e654187168cc595690bb42a5"} Nov 03 19:30:56.168157 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.165544 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:56.168157 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.165748 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:56.168157 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.165813 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs podName:90d8527c-7221-45b1-b2c5-9397fa8db4e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:58.165793354 +0000 UTC m=+6.030795795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs") pod "network-metrics-daemon-bbtrq" (UID: "90d8527c-7221-45b1-b2c5-9397fa8db4e3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:56.266858 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.266818 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:56.267049 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.266979 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:56.267049 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.267049 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret podName:7bac36d0-ca54-4a9c-83cd-c58ba9f58c37 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:58.267029729 +0000 UTC m=+6.132032181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret") pod "global-pull-secret-syncer-8z8g9" (UID: "7bac36d0-ca54-4a9c-83cd-c58ba9f58c37") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:56.367401 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.367323 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:56.367581 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.367563 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:30:56.367648 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.367593 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:30:56.367648 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.367606 2584 projected.go:196] Error preparing data for projected volume kube-api-access-x74s2 for pod openshift-network-diagnostics/network-check-target-pp285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:56.367780 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.367683 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2 podName:656bc97e-5a5b-4334-8369-bc842712cf02 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:58.36766311 +0000 UTC m=+6.232665542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x74s2" (UniqueName: "kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2") pod "network-check-target-pp285" (UID: "656bc97e-5a5b-4334-8369-bc842712cf02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:56.567683 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.564712 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:54 +0000 UTC" deadline="2028-08-04 07:59:30.720083304 +0000 UTC" Nov 03 19:30:56.567683 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.564771 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15348h28m34.155316179s" Nov 03 19:30:56.748077 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.746775 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:56.748077 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.746918 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:30:56.749946 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.748808 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:56.749946 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:56.748943 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:30:56.850369 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:56.850328 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"70af56b747e8dbbb62b5b41a5fc7a4ec","Type":"ContainerStarted","Data":"97480ce46a668e43467d52aef3b48208b56742283e9d3134eba41c60367a7040"} Nov 03 19:30:57.743397 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:57.743108 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:57.743832 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:57.743465 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:30:57.867595 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:57.867550 2584 generic.go:356] "Generic (PLEG): container finished" podID="300bf0c34ec8efd4851a1973652985e3" containerID="dae363d801e26fa6dbd3eb3a6dffa7851f0f9d7f4fc415d5b0fbe8110e30c218" exitCode=0 Nov 03 19:30:57.868719 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:57.868457 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"300bf0c34ec8efd4851a1973652985e3","Type":"ContainerDied","Data":"dae363d801e26fa6dbd3eb3a6dffa7851f0f9d7f4fc415d5b0fbe8110e30c218"} Nov 03 19:30:57.882342 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:57.881509 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" podStartSLOduration=3.881492991 podStartE2EDuration="3.881492991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:56.866142803 +0000 UTC m=+4.731145254" watchObservedRunningTime="2026-11-03 19:30:57.881492991 +0000 UTC m=+5.746495437" Nov 03 19:30:58.197730 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.197036 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:58.197730 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.197240 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:58.197730 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.197318 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs podName:90d8527c-7221-45b1-b2c5-9397fa8db4e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:02.197298033 +0000 UTC m=+10.062300460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs") pod "network-metrics-daemon-bbtrq" (UID: "90d8527c-7221-45b1-b2c5-9397fa8db4e3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:30:58.298889 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.298360 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:58.298889 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.298492 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:58.298889 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.298551 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret podName:7bac36d0-ca54-4a9c-83cd-c58ba9f58c37 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:02.298533533 +0000 UTC m=+10.163535971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret") pod "global-pull-secret-syncer-8z8g9" (UID: "7bac36d0-ca54-4a9c-83cd-c58ba9f58c37") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:30:58.399516 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.398941 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:58.399516 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.399106 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:30:58.399516 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.399127 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:30:58.399516 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.399138 2584 projected.go:196] Error preparing data for projected volume kube-api-access-x74s2 for pod openshift-network-diagnostics/network-check-target-pp285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:58.399516 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.399184 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2 podName:656bc97e-5a5b-4334-8369-bc842712cf02 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:02.399168105 +0000 UTC m=+10.264170542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x74s2" (UniqueName: "kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2") pod "network-check-target-pp285" (UID: "656bc97e-5a5b-4334-8369-bc842712cf02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:30:58.748969 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.748282 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:30:58.748969 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.748407 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:30:58.748969 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.748779 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:30:58.748969 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:58.748870 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:30:58.872769 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.872732 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"300bf0c34ec8efd4851a1973652985e3","Type":"ContainerStarted","Data":"0c2e2cad3c00d597419183037a3c0bd3a380eb7df2f661df23c0212e8bca6c03"} Nov 03 19:30:58.880561 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.880514 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerDied","Data":"d940c9652647e7e79592dfd6e720ebd0aab26c44bf415521ade1438f6a2105c2"} Nov 03 19:30:58.880713 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.880401 2584 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="d940c9652647e7e79592dfd6e720ebd0aab26c44bf415521ade1438f6a2105c2" exitCode=0 Nov 03 19:30:58.887366 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:58.887316 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" podStartSLOduration=4.887302601 podStartE2EDuration="4.887302601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:58.886508576 +0000 UTC m=+6.751511022" watchObservedRunningTime="2026-11-03 19:30:58.887302601 +0000 UTC m=+6.752305048" Nov 03 19:30:59.742929 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:59.742898 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:30:59.743116 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:30:59.743029 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:30:59.886391 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:30:59.886364 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8cmvr" event={"ID":"2d56c5c2-1d76-490b-b853-3e8d15662fa5","Type":"ContainerStarted","Data":"c0a593d56ae2421b3593a9e0f077741e7e8a14515871f7197bb26e2f45f4e484"} Nov 03 19:31:00.743967 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:00.743927 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:00.744112 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:00.744062 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:00.746197 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:00.746175 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:00.746318 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:00.746300 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:00.903268 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:00.903171 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zj9jh" event={"ID":"3fbf00d3-e30f-4399-b770-6545f69f15ce","Type":"ContainerStarted","Data":"04a4d035b5766b82e432f0f7dee26cf13347f7960b2d944aa73c967ead306a22"} Nov 03 19:31:00.915417 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:00.915363 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8cmvr" podStartSLOduration=3.675784116 podStartE2EDuration="7.915346583s" totalImagesPullingTime="4.239562467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:59.899545143 +0000 UTC m=+7.764547606" watchObservedRunningTime="2026-11-03 19:31:00.915346583 +0000 UTC m=+8.780349030" Nov 03 19:31:00.916371 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:00.916330 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zj9jh" podStartSLOduration=2.907332372 podStartE2EDuration="7.916314712s" totalImagesPullingTime="5.00898234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:00.915730285 +0000 UTC m=+8.780732733" watchObservedRunningTime="2026-11-03 19:31:00.916314712 +0000 UTC m=+8.781317176" Nov 03 19:31:01.742947 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:01.742915 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:01.743091 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:01.743049 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:01.908086 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:01.908048 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" event={"ID":"9309b9f0-2a67-4c14-8af9-8759348ae5d5","Type":"ContainerStarted","Data":"36ea6056f19921c6923bb3690bf4d9b197c23bb9ab9cbb7390c17b0438c15928"} Nov 03 19:31:02.232441 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.231851 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:02.232441 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.231999 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:31:02.232441 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.232057 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs podName:90d8527c-7221-45b1-b2c5-9397fa8db4e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:10.232040348 +0000 UTC m=+18.097042775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs") pod "network-metrics-daemon-bbtrq" (UID: "90d8527c-7221-45b1-b2c5-9397fa8db4e3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:31:02.333855 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.333296 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:02.333855 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.333434 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:31:02.333855 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.333491 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret podName:7bac36d0-ca54-4a9c-83cd-c58ba9f58c37 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:10.333473667 +0000 UTC m=+18.198476094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret") pod "global-pull-secret-syncer-8z8g9" (UID: "7bac36d0-ca54-4a9c-83cd-c58ba9f58c37") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:31:02.433900 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.433863 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:02.434056 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.433998 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:31:02.434056 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.434022 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:31:02.434056 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.434034 2584 projected.go:196] Error preparing data for projected volume kube-api-access-x74s2 for pod openshift-network-diagnostics/network-check-target-pp285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:31:02.434217 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.434081 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2 podName:656bc97e-5a5b-4334-8369-bc842712cf02 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:10.434068728 +0000 UTC m=+18.299071156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x74s2" (UniqueName: "kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2") pod "network-check-target-pp285" (UID: "656bc97e-5a5b-4334-8369-bc842712cf02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:31:02.641785 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.641705 2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:31:02.744354 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.744318 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:02.744525 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.744453 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:02.744525 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.744472 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:02.744649 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:02.744590 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:02.820094 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.819918 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:02.820779 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.820520 2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:02.912854 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.912809 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" event={"ID":"9309b9f0-2a67-4c14-8af9-8759348ae5d5","Type":"ContainerStarted","Data":"a6bb9c94045df692a3c2d46229630ac3c85f5e3a9d5d7bc303119b64f226c0c5"} Nov 03 19:31:02.916348 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:02.916321 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6fjx" event={"ID":"d7f898aa-1924-459a-af1c-2242ac527e2a","Type":"ContainerStarted","Data":"f1dc5c5225eb371a495fed58e981ba31b16d91d45383b096cf9cbb18e26ee0ac"} Nov 03 19:31:03.604949 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.604820 2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:31:02.641727627Z","UUID":"3a5870c0-9896-4f5d-889a-625fd05005c9","Handler":null,"Name":"","Endpoint":""} Nov 03 19:31:03.606467 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.606445 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:31:03.606577 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.606478 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:31:03.742655 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.742631 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:03.742777 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:03.742742 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:03.920146 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.920112 2584 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="dfff7bfdc83388e91e370d3d88e8f20e98dd1e5d05d88ea50fc854989b84bd79" exitCode=0 Nov 03 19:31:03.920495 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.920190 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerDied","Data":"dfff7bfdc83388e91e370d3d88e8f20e98dd1e5d05d88ea50fc854989b84bd79"} Nov 03 19:31:03.922235 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.922180 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ns5zn" event={"ID":"0225e321-66c4-4269-81d4-051793673f4d","Type":"ContainerStarted","Data":"413bd2b4316b6e0c9d252353c14c32667a7e7ac1f30d37dca6e2819c29775103"} Nov 03 19:31:03.925513 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.925486 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" event={"ID":"9309b9f0-2a67-4c14-8af9-8759348ae5d5","Type":"ContainerStarted","Data":"60198eebca05b761b56c8714d1a12fa2629882dd906a92d8af212ae16630b6a9"} Nov 03 19:31:03.940736 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.940686 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-k6fjx" podStartSLOduration=3.211252565 podStartE2EDuration="10.940671874s" totalImagesPullingTime="7.729419309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:02.931518526 +0000 UTC m=+10.796520975" watchObservedRunningTime="2026-11-03 19:31:03.940671874 +0000 UTC m=+11.805674324" Nov 03 19:31:03.955179 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.955109 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" podStartSLOduration=2.515889275 podStartE2EDuration="10.955097291s" totalImagesPullingTime="8.439208016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:03.954551021 +0000 UTC m=+11.819553489" watchObservedRunningTime="2026-11-03 19:31:03.955097291 +0000 UTC m=+11.820099738" Nov 03 19:31:03.971126 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:03.971088 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ns5zn" podStartSLOduration=2.905865864 podStartE2EDuration="10.971079145s" totalImagesPullingTime="8.065213281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:03.971039609 +0000 UTC m=+11.836042056" watchObservedRunningTime="2026-11-03 19:31:03.971079145 +0000 UTC m=+11.836081594" Nov 03 19:31:04.744922 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:04.744711 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:04.745083 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:04.745043 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:04.745516 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:04.745492 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:04.745646 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:04.745624 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:04.928843 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:04.928757 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-629fw" event={"ID":"4e4278cb-b8ca-421a-804b-dc7c728ba346","Type":"ContainerStarted","Data":"a42402b730f607d8f882b89d1a7bd25b51ea634972d939039e55df546f84c473"} Nov 03 19:31:04.932353 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:04.932327 2584 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="7876dc84bf2a66a9b2aff2a4a8b63a302a85ef603598baf021dc130da50fa65a" exitCode=0 Nov 03 19:31:04.932477 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:04.932446 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerDied","Data":"7876dc84bf2a66a9b2aff2a4a8b63a302a85ef603598baf021dc130da50fa65a"} Nov 03 19:31:04.942154 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:04.942105 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-629fw" podStartSLOduration=4.315139449 podStartE2EDuration="11.942089634s" totalImagesPullingTime="7.626950185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:04.941729075 +0000 UTC m=+12.806731523" watchObservedRunningTime="2026-11-03 19:31:04.942089634 +0000 UTC m=+12.807092081" Nov 03 19:31:05.743531 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:05.743496 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:05.743719 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:05.743643 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:05.936097 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:05.936062 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" event={"ID":"e17fbd17-d288-4823-bad6-cdd6d37e5c15","Type":"ContainerStarted","Data":"0564c1c7a28d9264212867f2b48ca883ae97dd5b965ee4f6d58664c0d96a3493"} Nov 03 19:31:05.951903 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:05.951846 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" podStartSLOduration=2.673340812 podStartE2EDuration="12.951829079s" totalImagesPullingTime="10.278488267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:05.951509205 +0000 UTC m=+13.816511653" watchObservedRunningTime="2026-11-03 19:31:05.951829079 +0000 UTC m=+13.816831529" Nov 03 19:31:06.743464 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:06.743432 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:06.743623 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:06.743547 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:06.743623 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:06.743604 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:06.743748 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:06.743723 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:06.941088 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:06.941053 2584 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="de46499ea63a40f0b7f94a761163a5e86e98111a8cea40e72d5d85140b47c56d" exitCode=0 Nov 03 19:31:06.941913 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:06.941095 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerDied","Data":"de46499ea63a40f0b7f94a761163a5e86e98111a8cea40e72d5d85140b47c56d"} Nov 03 19:31:07.743133 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:07.743101 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:07.743316 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:07.743209 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:08.743379 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:08.743351 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:08.743796 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:08.743458 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:08.743859 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:08.743836 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:08.743984 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:08.743912 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:09.742720 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.742559 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:09.742810 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:09.742788 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:09.949389 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.949367 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:09.949792 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.949728 2584 generic.go:356] "Generic (PLEG): container finished" podID="c8b6db40-add7-4eca-9480-e174e021fdc7" containerID="6118976b097e9e300e55553fc2dee54187c5da1bf122c99c828d6087e62e71ee" exitCode=1 Nov 03 19:31:09.949853 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.949792 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"ae37b98d9ef10ee7126cb297bc3f2d8ce456b6db5e6fa9867f25545d8f91a4ca"} Nov 03 19:31:09.949887 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.949863 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"3d6cbceb6fe3c747f9cc7c47e15ffee8d81c293c719396bd4efef65a7a9da4c2"} Nov 03 19:31:09.949921 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.949911 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerDied","Data":"6118976b097e9e300e55553fc2dee54187c5da1bf122c99c828d6087e62e71ee"} Nov 03 19:31:09.949970 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.949958 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"591e8198753ab37ff514062f2bec4b3b1d8185e967f1275ba3846c83f78ea709"} Nov 03 19:31:09.952753 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.952732 2584 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="59cf718db877ac07bd14fd4f81cbad79976f6966b0675886be5a5bcfc2092e59" exitCode=0 Nov 03 19:31:09.952852 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:09.952759 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerDied","Data":"59cf718db877ac07bd14fd4f81cbad79976f6966b0675886be5a5bcfc2092e59"} Nov 03 19:31:10.292739 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.292494 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:10.292739 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.292664 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:31:10.292915 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.292782 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs podName:90d8527c-7221-45b1-b2c5-9397fa8db4e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:26.292765348 +0000 UTC m=+34.157767789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs") pod "network-metrics-daemon-bbtrq" (UID: "90d8527c-7221-45b1-b2c5-9397fa8db4e3") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:31:10.392975 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.392944 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:10.393101 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.393081 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:31:10.393146 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.393136 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret podName:7bac36d0-ca54-4a9c-83cd-c58ba9f58c37 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:26.393123447 +0000 UTC m=+34.258125871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret") pod "global-pull-secret-syncer-8z8g9" (UID: "7bac36d0-ca54-4a9c-83cd-c58ba9f58c37") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:31:10.493446 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.493409 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:10.493619 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.493594 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:31:10.493619 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.493616 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:31:10.493718 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.493629 2584 projected.go:196] Error preparing data for projected volume kube-api-access-x74s2 for pod openshift-network-diagnostics/network-check-target-pp285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:31:10.493718 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.493683 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2 podName:656bc97e-5a5b-4334-8369-bc842712cf02 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:26.493666002 +0000 UTC m=+34.358668439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x74s2" (UniqueName: "kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2") pod "network-check-target-pp285" (UID: "656bc97e-5a5b-4334-8369-bc842712cf02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:31:10.678090 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.678016 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:10.678214 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.678143 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:31:10.678677 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.678661 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:10.743329 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.743308 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:10.743422 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.743407 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:10.743512 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.743494 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:10.743611 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:10.743577 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:10.957220 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.957157 2584 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="6e5a017536a472d02d7817180f8ae2e33c03293bac513ae9e239a5426bf2c35e" exitCode=0 Nov 03 19:31:10.957650 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.957244 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerDied","Data":"6e5a017536a472d02d7817180f8ae2e33c03293bac513ae9e239a5426bf2c35e"} Nov 03 19:31:10.959997 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.959980 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:10.960340 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.960323 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"a68627e8a3894323fec52ab4849e7230ce0d1af6270ad7921688faebe3a9d363"} Nov 03 19:31:10.960401 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:10.960376 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"ba81faccad39aa39e83ce4c728b2db20ed2ef13efd03e5eb59dc739b3789f0cd"} Nov 03 19:31:11.742800 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:11.742762 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:11.742982 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:11.742863 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:11.965665 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:11.965620 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" event={"ID":"8bf911cd-cd3c-4990-81ce-80d9ef41f214","Type":"ContainerStarted","Data":"c65341ac6d3517552ecba06250639b8a5e535e31cd3a0d06dbbcab40a32b5bf1"} Nov 03 19:31:11.988307 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:11.988270 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jf9qv" podStartSLOduration=7.815924866 podStartE2EDuration="18.988258257s" totalImagesPullingTime="11.172333391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:11.987972755 +0000 UTC m=+19.852975203" watchObservedRunningTime="2026-11-03 19:31:11.988258257 +0000 UTC m=+19.853260702" Nov 03 19:31:12.743893 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:12.743848 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:12.744089 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:12.743949 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:12.744089 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:12.744037 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:12.744201 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:12.744137 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:12.971058 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:12.971033 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:12.971597 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:12.971455 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"c26817dcc4318d4e30d12c862da557b668374f1aa02cd1d448f2ca0eed3c8119"} Nov 03 19:31:13.743339 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:13.743299 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:13.743513 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:13.743399 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:14.743262 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:14.743228 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:14.743887 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:14.743364 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:14.743887 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:14.743416 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:14.743887 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:14.743508 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:14.978427 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:14.978404 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:14.978803 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:14.978782 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"e1fa8c4a7d2f601e43859d08f576ea7960bdc40f41edbd803e545da9998f01c9"} Nov 03 19:31:14.979018 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:14.979007 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:14.979124 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:14.979109 2584 scope.go:122] "RemoveContainer" containerID="6118976b097e9e300e55553fc2dee54187c5da1bf122c99c828d6087e62e71ee" Nov 03 19:31:14.995066 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:14.995020 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:15.743114 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.742962 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:15.743256 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:15.743206 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:15.923546 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.923459 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8z8g9"] Nov 03 19:31:15.923893 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.923560 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:15.923893 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:15.923658 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:15.926806 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.926732 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pp285"] Nov 03 19:31:15.927470 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.927448 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bbtrq"] Nov 03 19:31:15.927596 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.927563 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:15.927737 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:15.927704 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:15.984983 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.984958 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:15.985331 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.985309 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" event={"ID":"c8b6db40-add7-4eca-9480-e174e021fdc7","Type":"ContainerStarted","Data":"414292f524079412f7503071b0fddbdacf882e2d71490287d6b0b464399ecc85"} Nov 03 19:31:15.985385 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.985319 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:15.985468 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:15.985449 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:15.985468 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.985375 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:31:15.985540 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:15.985478 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:16.000908 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:16.000889 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:16.008159 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:16.008122 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" podStartSLOduration=8.193776486 podStartE2EDuration="23.008109803s" totalImagesPullingTime="14.814333317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:16.007406352 +0000 UTC m=+23.872408797" watchObservedRunningTime="2026-11-03 19:31:16.008109803 +0000 UTC m=+23.873112249" Nov 03 19:31:16.987798 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:16.987772 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:31:17.743358 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:17.743325 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:17.743550 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:17.743389 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:17.743550 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:17.743474 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:17.743550 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:17.743482 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:17.743661 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:17.743549 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:17.743661 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:17.743624 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:17.989544 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:17.989517 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:31:19.742944 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:19.742914 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:19.743428 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:19.742914 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:19.743428 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:19.743026 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:19.743428 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:19.742914 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:19.743428 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:19.743091 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:19.743428 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:19.743172 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:19.796484 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:19.796459 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:19.796643 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:19.796628 2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:31:19.806222 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:19.806196 2584 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" podUID="c8b6db40-add7-4eca-9480-e174e021fdc7" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 19:31:19.814421 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:19.814396 2584 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" podUID="c8b6db40-add7-4eca-9480-e174e021fdc7" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 19:31:21.742802 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:21.742775 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:21.743185 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:21.742838 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:21.743185 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:21.742863 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:21.743185 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:21.742951 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bbtrq" podUID="90d8527c-7221-45b1-b2c5-9397fa8db4e3" Nov 03 19:31:21.743185 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:21.742986 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8z8g9" podUID="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" Nov 03 19:31:21.743185 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:21.743048 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp285" podUID="656bc97e-5a5b-4334-8369-bc842712cf02" Nov 03 19:31:22.406481 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:22.406421 2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeReady" Nov 03 19:31:22.406625 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:22.406519 2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 19:31:23.743250 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.743216 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:23.743674 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.743283 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:23.743674 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.743364 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:23.746689 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.746665 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:31:23.746689 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.746687 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:31:23.746930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.746722 2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:31:23.746930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.746906 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:31:23.746930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.746906 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77jlf\"" Nov 03 19:31:23.746930 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.746919 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khl78\"" Nov 03 19:31:23.747116 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:23.746928 2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tdq2h\"" Nov 03 19:31:26.308337 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.308308 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:26.308725 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:26.308409 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:31:26.308725 ip-10-0-133-204 kubenswrapper[2584]: E1103 19:31:26.308465 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs podName:90d8527c-7221-45b1-b2c5-9397fa8db4e3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:58.308452635 +0000 UTC m=+66.173455059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs") pod "network-metrics-daemon-bbtrq" (UID: "90d8527c-7221-45b1-b2c5-9397fa8db4e3") : secret "metrics-daemon-secret" not found Nov 03 19:31:26.408688 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.408661 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:26.411979 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.411958 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:26.508943 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.508870 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:26.511344 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.511321 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:26.525966 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.525947 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:26.538086 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.538063 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:26.938322 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.938251 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pp285"] Nov 03 19:31:26.941408 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:26.941382 2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8z8g9"] Nov 03 19:31:26.942617 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:31:26.942593 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656bc97e_5a5b_4334_8369_bc842712cf02.slice/crio-b71a2dfa0b931fa39b5ca0d882ef0b1da3a7a5d7c6e71ebd59a029a242edc1d0 WatchSource:0}: Error finding container b71a2dfa0b931fa39b5ca0d882ef0b1da3a7a5d7c6e71ebd59a029a242edc1d0: Status 404 returned error can't find the container with id b71a2dfa0b931fa39b5ca0d882ef0b1da3a7a5d7c6e71ebd59a029a242edc1d0 Nov 03 19:31:26.944968 ip-10-0-133-204 kubenswrapper[2584]: W1103 19:31:26.944948 2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bac36d0_ca54_4a9c_83cd_c58ba9f58c37.slice/crio-e10db87b493b86eb13afb4b3cf43392f57e8b52fb39fee404bdab1f2265aaee4 WatchSource:0}: Error finding container e10db87b493b86eb13afb4b3cf43392f57e8b52fb39fee404bdab1f2265aaee4: Status 404 returned error can't find the container with id e10db87b493b86eb13afb4b3cf43392f57e8b52fb39fee404bdab1f2265aaee4 Nov 03 19:31:27.007958 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:27.007932 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8z8g9" event={"ID":"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37","Type":"ContainerStarted","Data":"e10db87b493b86eb13afb4b3cf43392f57e8b52fb39fee404bdab1f2265aaee4"} Nov 03 19:31:27.008942 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:27.008921 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pp285" event={"ID":"656bc97e-5a5b-4334-8369-bc842712cf02","Type":"ContainerStarted","Data":"b71a2dfa0b931fa39b5ca0d882ef0b1da3a7a5d7c6e71ebd59a029a242edc1d0"} Nov 03 19:31:27.231045 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:27.230981 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cmvr_2d56c5c2-1d76-490b-b853-3e8d15662fa5/konnectivity-agent/0.log" limit=1024 Nov 03 19:31:27.617444 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:27.617412 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8cmvr_2d56c5c2-1d76-490b-b853-3e8d15662fa5/konnectivity-agent/0.log" limit=1024 Nov 03 19:31:27.678454 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:27.678424 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ns5zn_0225e321-66c4-4269-81d4-051793673f4d/dns-node-resolver/0.log" Nov 03 19:31:29.079679 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:29.079646 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zj9jh_3fbf00d3-e30f-4399-b770-6545f69f15ce/node-ca/0.log" Nov 03 19:31:31.020575 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:31.020537 2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pp285" event={"ID":"656bc97e-5a5b-4334-8369-bc842712cf02","Type":"ContainerStarted","Data":"0e73a12b6dc721c4dd190f6deb9a9d76becee79836b914c97acff76947dadf45"} Nov 03 19:31:31.021035 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:31.020695 2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:31.033168 ip-10-0-133-204 kubenswrapper[2584]: I1103 19:31:31.033125 2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pp285" podStartSLOduration=34.373116584 podStartE2EDuration="38.033113776s" totalImagesPullingTime="3.659997192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:31.032740089 +0000 UTC m=+38.897742536" watchObservedRunningTime="2026-11-03 19:31:31.033113776 +0000 UTC m=+38.898116222" Nov 03 19:31:32.236777 ip-10-0-133-204 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 19:31:32.255053 ip-10-0-133-204 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 19:31:32.255298 ip-10-0-133-204 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 19:31:32.255362 ip-10-0-133-204 systemd[1]: kubelet.service: Consumed 2.041s CPU time. Nov 03 19:31:32.267143 ip-10-0-133-204 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:31:32.349449 ip-10-0-133-204 kubenswrapper[4314]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:31:32.349449 ip-10-0-133-204 kubenswrapper[4314]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:31:32.349449 ip-10-0-133-204 kubenswrapper[4314]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:31:32.349449 ip-10-0-133-204 kubenswrapper[4314]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:31:32.351537 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351458 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.351537 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351537 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351541 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351544 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351548 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351550 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351553 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351557 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351560 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351567 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351570 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351572 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351575 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351592 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351595 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351598 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351600 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351603 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351606 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351608 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.351633 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351611 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351616 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351620 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351624 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351627 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351630 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351633 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351636 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351639 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351642 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351645 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351648 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351650 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351653 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351655 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351658 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351661 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351663 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351666 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.352135 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351668 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351671 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351674 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351676 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351679 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351682 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351684 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351687 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351690 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351693 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351696 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351699 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351702 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351704 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351707 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351711 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351713 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351716 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351720 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.352611 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351724 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351727 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351730 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351733 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351736 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351738 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351741 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351743 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351746 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351749 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351751 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351754 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351756 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351759 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351762 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351765 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351768 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351770 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351773 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351775 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.353091 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351778 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351781 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351784 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351787 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351789 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351792 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351794 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351797 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351799 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351802 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351806 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351808 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351811 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351813 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351816 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351818 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351821 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351823 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351826 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351828 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.353621 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351831 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351833 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351836 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351838 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351841 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351843 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351846 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.351849 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352605 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352612 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352615 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352618 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352620 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352625 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352628 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352630 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352633 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352636 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352638 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.354121 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352641 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352644 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352646 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352649 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352652 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352654 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352657 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352660 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352662 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352665 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352667 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352670 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352673 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352675 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352678 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352680 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352683 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352685 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352688 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.354609 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352691 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352693 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352703 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352707 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352709 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352712 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352715 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352718 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352720 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352723 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352726 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352728 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352731 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352733 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352736 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352739 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352741 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352744 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352746 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352749 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.355088 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352751 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352754 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352756 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352759 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352761 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352764 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352766 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352769 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352774 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352778 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352782 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352785 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352787 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352789 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352792 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352795 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352798 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352801 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352803 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352806 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.355593 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352809 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352811 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352814 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352816 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352819 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352821 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352824 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352826 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352829 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352831 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352835 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352837 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352839 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352842 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352844 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352848 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352852 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352855 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352858 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.356080 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352861 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352864 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352866 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352869 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352872 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352874 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352877 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352880 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352882 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352885 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352887 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352890 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352893 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352896 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352899 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352901 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.352903 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353352 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353357 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353360 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.356544 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353363 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353366 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353370 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353374 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353378 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353381 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353384 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353386 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353390 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353393 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353395 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353398 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353401 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353404 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353406 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353409 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353412 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353415 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353417 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.357085 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353420 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353423 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353426 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353428 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353431 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353433 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353436 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353438 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353440 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353443 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353446 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353448 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353450 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353453 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353456 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353458 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353461 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353463 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353466 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353469 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.357602 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353471 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353474 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353476 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353479 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353481 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353483 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353486 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353489 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353491 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353494 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353496 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353499 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353501 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353504 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353507 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353510 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353515 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353518 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353521 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353524 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.358101 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353526 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353529 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353532 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353534 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353537 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353539 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353542 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353544 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353547 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353550 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353553 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353555 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353558 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353560 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353563 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353565 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353568 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353571 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353574 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353576 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.358661 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353592 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353595 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353598 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353600 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353603 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353605 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353608 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353611 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353614 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353616 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353619 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353622 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353625 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353628 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353631 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353634 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353636 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353639 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353642 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.359452 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353644 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353648 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353650 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353653 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.353656 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353752 4314 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353760 4314 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353772 4314 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353779 4314 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353786 4314 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353791 4314 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353803 4314 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353808 4314 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353811 4314 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353814 4314 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353818 4314 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353822 4314 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353825 4314 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353828 4314 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353831 4314 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353834 4314 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353837 4314 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353841 4314 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:31:32.360224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353845 4314 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353848 4314 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353851 4314 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353855 4314 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353858 4314 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353862 4314 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353865 4314 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353869 4314 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353874 4314 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353878 4314 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353881 4314 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353884 4314 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353887 4314 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353890 4314 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353894 4314 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353898 4314 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353900 4314 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353904 4314 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353909 4314 flags.go:64] FLAG: --enable-server="true" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353912 4314 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353917 4314 flags.go:64] FLAG: --event-burst="100" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353920 4314 flags.go:64] FLAG: --event-qps="50" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353923 4314 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353926 4314 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:31:32.361270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353929 4314 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353934 4314 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353936 4314 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353939 4314 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353943 4314 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353946 4314 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353949 4314 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353952 4314 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353955 4314 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353958 4314 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353961 4314 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353964 4314 flags.go:64] FLAG: --feature-gates="" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353968 4314 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353971 4314 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353974 4314 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353977 4314 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353982 4314 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353985 4314 flags.go:64] FLAG: --help="false" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353988 4314 flags.go:64] FLAG: --hostname-override="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353991 4314 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353995 4314 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.353998 4314 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354001 4314 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:31:32.362353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354005 4314 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354008 4314 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354011 4314 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354015 4314 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354018 4314 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354021 4314 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354024 4314 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354027 4314 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354030 4314 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354033 4314 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354036 4314 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354039 4314 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354042 4314 flags.go:64] FLAG: --lock-file="" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354044 4314 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354047 4314 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354051 4314 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354056 4314 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354059 4314 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354063 4314 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354066 4314 flags.go:64] FLAG: --logging-format="text" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354069 4314 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354072 4314 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354075 4314 flags.go:64] FLAG: --manifest-url="" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354078 4314 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354082 4314 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:31:32.363362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354087 4314 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354091 4314 flags.go:64] FLAG: --max-pods="110" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354094 4314 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354097 4314 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354100 4314 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354103 4314 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354106 4314 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354109 4314 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354112 4314 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354119 4314 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354124 4314 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354127 4314 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354130 4314 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354133 4314 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354136 4314 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354139 4314 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354142 4314 flags.go:64] FLAG: --port="10250" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354145 4314 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354148 4314 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0bbb2aae223ccbeb8" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354151 4314 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354154 4314 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354157 4314 flags.go:64] FLAG: --register-node="true" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354160 4314 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354164 4314 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354167 4314 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354170 4314 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:31:32.364461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354173 4314 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354177 4314 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354179 4314 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354182 4314 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354185 4314 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354188 4314 flags.go:64] FLAG: --runonce="false" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354196 4314 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354199 4314 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354202 4314 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354205 4314 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354208 4314 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354211 4314 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354214 4314 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354217 4314 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354220 4314 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354223 4314 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354227 4314 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354230 4314 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354233 4314 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354236 4314 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354239 4314 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354244 4314 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354247 4314 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354250 4314 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354254 4314 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:31:32.365652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354257 4314 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354260 4314 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354263 4314 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354266 4314 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354269 4314 flags.go:64] FLAG: --v="2" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354273 4314 flags.go:64] FLAG: --version="false" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354277 4314 flags.go:64] FLAG: --vmodule="" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354281 4314 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354284 4314 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354424 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354428 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354431 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354434 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354438 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354442 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354445 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354449 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354452 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354455 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354457 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354460 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.366610 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354463 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354465 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354469 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354472 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354475 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354477 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354480 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354483 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354486 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354488 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354491 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354493 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354496 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354499 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354501 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354504 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354506 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354509 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354511 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354514 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.367241 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354523 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354526 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354529 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354531 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354535 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354538 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354541 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354544 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354546 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354549 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354552 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354554 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354556 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354559 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354563 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354565 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354568 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354571 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354573 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354576 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.367805 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354593 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354596 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354598 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354601 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354604 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354606 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354609 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354612 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354615 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354618 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354621 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354623 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354626 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354637 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354640 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354643 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354647 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354650 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354652 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.368365 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354655 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354657 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354660 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354663 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354665 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354668 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354671 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354674 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354677 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354680 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354683 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354685 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354688 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354691 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354693 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354696 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354699 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354701 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354704 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.368994 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354706 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354709 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354712 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354714 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354717 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354719 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354722 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354724 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354733 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354735 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354740 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354744 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354747 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354750 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354753 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354755 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.369741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.354782 4314 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354826 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354832 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354836 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354839 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354843 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354846 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354849 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354851 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354854 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354856 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354859 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354862 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354864 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354867 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354869 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354872 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354875 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354877 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.370366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354880 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354883 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354885 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354888 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354890 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354893 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354896 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354900 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354902 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354906 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354908 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354911 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354913 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354916 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354918 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354921 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354924 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354926 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354930 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354933 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.370858 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354935 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354938 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354940 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354943 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354945 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354948 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354954 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354957 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354959 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354962 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354964 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354967 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354969 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354972 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354974 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354977 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354979 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354982 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354985 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.371374 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354995 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.354998 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355001 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355003 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355013 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355015 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355018 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355021 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355023 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355026 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355028 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355033 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355035 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355038 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355041 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355044 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355046 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355049 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355051 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355054 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.371896 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355057 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355060 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355063 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355065 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355068 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355071 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355073 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355076 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355078 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355081 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355083 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355086 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355090 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355098 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355101 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355104 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355106 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355109 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355112 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.372511 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355114 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355119 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355122 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355126 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355130 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355132 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355135 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355137 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355140 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.355142 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.359824 4314 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.359844 4314 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.359869 4314 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.359878 4314 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.359969 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.359977 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.359983 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.359988 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.359993 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.373202 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.359997 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360002 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360006 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360011 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360015 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360020 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360024 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360029 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360033 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360037 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360042 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360049 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360053 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360057 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360062 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360066 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360071 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360075 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360080 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360085 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.373678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360089 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360093 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360098 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360102 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360106 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360111 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360116 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360121 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360126 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360130 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360134 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360138 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360143 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360147 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360151 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360155 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360160 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360164 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360168 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.374203 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360173 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360177 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360181 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360186 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360191 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360195 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360199 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360203 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360208 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360212 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360216 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360220 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360224 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360228 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360233 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360237 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360241 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360245 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360249 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360253 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.374679 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360260 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360268 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360274 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360279 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360284 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360288 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360293 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360297 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360301 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360306 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360311 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360315 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360320 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360324 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360328 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360334 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360338 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360343 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360347 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.375154 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360352 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360356 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360361 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360365 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360369 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360374 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360378 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360382 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360387 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360391 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360395 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360400 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360404 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360408 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360413 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360417 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360422 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360427 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360431 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.375618 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360435 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360439 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360444 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360448 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.360490 4314 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360730 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360739 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360744 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360748 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360752 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360758 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360762 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360767 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360772 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360777 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360781 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.376071 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360785 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360790 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360794 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360798 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360802 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360807 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360812 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360816 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360821 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360825 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360829 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360834 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360838 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360842 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360846 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360851 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360855 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360859 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360864 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.376451 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360868 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360872 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360876 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360881 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360885 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360889 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360893 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360897 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360902 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360908 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360912 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360917 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360921 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360925 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360929 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360934 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360938 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360942 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360946 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.376952 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360951 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360957 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360963 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360968 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360972 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360977 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360981 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360986 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360990 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360994 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.360998 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361002 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361007 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361011 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361015 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361019 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361023 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361028 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361032 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.377460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361036 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361041 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361045 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361051 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361055 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361059 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361063 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361068 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361072 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361077 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361081 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361085 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361089 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361093 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361097 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361101 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361106 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361110 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361114 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361119 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.378055 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361125 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361131 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361136 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361142 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361147 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361151 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361156 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361165 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361170 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361175 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361180 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361184 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361189 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361193 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361198 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361203 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361207 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.378872 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361212 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.361249 4314 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361299 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361307 4314 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361312 4314 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361317 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361322 4314 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361326 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361331 4314 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361335 4314 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361339 4314 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361344 4314 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361348 4314 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361353 4314 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361357 4314 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361361 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:31:32.379523 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361366 4314 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361370 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361374 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361379 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361383 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361388 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361392 4314 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361399 4314 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361403 4314 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361408 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361412 4314 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361416 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361421 4314 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361425 4314 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361431 4314 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361437 4314 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361443 4314 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361449 4314 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361455 4314 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:31:32.380160 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361459 4314 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361463 4314 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361468 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361472 4314 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361476 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361480 4314 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361484 4314 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361489 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361493 4314 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361498 4314 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361502 4314 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361506 4314 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361510 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361515 4314 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361519 4314 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361523 4314 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361527 4314 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361531 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361536 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361540 4314 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:31:32.380708 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361549 4314 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361554 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361558 4314 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361563 4314 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361567 4314 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361571 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361576 4314 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361596 4314 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361600 4314 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361605 4314 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361610 4314 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361615 4314 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361620 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361627 4314 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361633 4314 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361638 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361643 4314 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361649 4314 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361653 4314 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361658 4314 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:31:32.381185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361662 4314 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361666 4314 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361671 4314 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361675 4314 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361679 4314 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361683 4314 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361688 4314 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361692 4314 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361697 4314 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361701 4314 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361705 4314 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361709 4314 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361716 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361720 4314 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361724 4314 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361729 4314 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361733 4314 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361738 4314 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361742 4314 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361747 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:31:32.381669 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361751 4314 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361756 4314 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361760 4314 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361765 4314 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361770 4314 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361774 4314 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361778 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361783 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361788 4314 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361792 4314 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361805 4314 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361810 4314 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:32.361814 4314 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.361901 4314 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.363408 4314 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.363476 4314 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.363794 4314 server.go:1014] "Starting client certificate rotation" Nov 03 19:31:32.382167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.363921 4314 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.363974 4314 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:25:52 +0000 UTC" deadline="2028-07-15 11:58:11.967388984 +0000 UTC" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.364006 4314 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14872h26m39.603384433s" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.364499 4314 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.366128 4314 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.367641 4314 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.369062 4314 log.go:25] "Validated CRI v1 image API" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.369824 4314 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:31:32.382651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.371215 4314 fs.go:135] Filesystem UUIDs: map[0909f9ba-24bc-4eca-8b36-eb44a5b68d76:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d5567e3c-870c-4519-8921-8cf21cb58727:/dev/nvme0n1p3] Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.371231 4314 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cbe6aa8bcb1b8a94796ba0f2d28fc89e35cbd8e5f0bcc75d46dde2430abc478/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cbe6aa8bcb1b8a94796ba0f2d28fc89e35cbd8e5f0bcc75d46dde2430abc478/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83c3e661343f8afd2c02edc813790851a08b88c0f168ba1f7b251012ed1bdbc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83c3e661343f8afd2c02edc813790851a08b88c0f168ba1f7b251012ed1bdbc8/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/957407ed50e65883014afa775cc085e9500e73441a343a28f2228df26942aaac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/957407ed50e65883014afa775cc085e9500e73441a343a28f2228df26942aaac/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9aabebd259389a1a32ec28a8ebe43dd4ee6f51d0a1c4dd24de6199f9e7dabef7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9aabebd259389a1a32ec28a8ebe43dd4ee6f51d0a1c4dd24de6199f9e7dabef7/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a69fa057d93699cf33a3f8219fd1adb86b4d4c02f763e9532e1f4c6137ae1e2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a69fa057d93699cf33a3f8219fd1adb86b4d4c02f763e9532e1f4c6137ae1e2e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b71a2dfa0b931fa39b5ca0d882ef0b1da3a7a5d7c6e71ebd59a029a242edc1d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b71a2dfa0b931fa39b5ca0d882ef0b1da3a7a5d7c6e71ebd59a029a242edc1d0/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9e625b9dd71b93087b9217529781377464a33122fe62cd15c87d02677fdebd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9e625b9dd71b93087b9217529781377464a33122fe62cd15c87d02677fdebd9/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3b8a09106b087bfa2f77220293563f980f76ac6e654187168cc595690bb42a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3b8a09106b087bfa2f77220293563f980f76ac6e654187168cc595690bb42a5/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e10db87b493b86eb13afb4b3cf43392f57e8b52fb39fee404bdab1f2265aaee4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e10db87b493b86eb13afb4b3cf43392f57e8b52fb39fee404bdab1f2265aaee4/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebf57f6c0b543b94e8a7226744d3ac9a460f414a560a799b9f092954c3975b05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebf57f6c0b543b94e8a7226744d3ac9a460f414a560a799b9f092954c3975b05/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0225e321-66c4-4269-81d4-051793673f4d/volumes/kubernetes.io~projected/kube-api-access-fw6qm:{mountpoint:/var/lib/kubelet/pods/0225e321-66c4-4269-81d4-051793673f4d/volumes/kubernetes.io~projected/kube-api-access-fw6qm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d56c5c2-1d76-490b-b853-3e8d15662fa5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2d56c5c2-1d76-490b-b853-3e8d15662fa5/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fbf00d3-e30f-4399-b770-6545f69f15ce/volumes/kubernetes.io~projected/kube-api-access-nb56k:{mountpoint:/var/lib/kubelet/pods/3fbf00d3-e30f-4399-b770-6545f69f15ce/volumes/kubernetes.io~projected/kube-api-access-nb56k major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4278cb-b8ca-421a-804b-dc7c728ba346/volumes/kubernetes.io~projected/kube-api-access-d2zkv:{mountpoint:/var/lib/kubelet/pods/4e4278cb-b8ca-421a-804b-dc7c728ba346/volumes/kubernetes.io~projected/kube-api-access-d2zkv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/656bc97e-5a5b-4334-8369-bc842712cf02/volumes/kubernetes.io~projected/kube-api-access-x74s2:{mountpoint:/var/lib/kubelet/pods/656bc97e-5a5b-4334-8369-bc842712cf02/volumes/kubernetes.io~projected/kube-api-access-x74s2 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bf911cd-cd3c-4990-81ce-80d9ef41f214/volumes/kubernetes.io~projected/kube-api-access-t5q4r:{mountpoint:/var/lib/kubelet/pods/8bf911cd-cd3c-4990-81ce-80d9ef41f214/volumes/kubernetes.io~projected/kube-api-access-t5q4r major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90d8527c-7221-45b1-b2c5-9397fa8db4e3/volumes/kubernetes.io~projected/kube-api-access-8k8zx:{mountpoint:/var/lib/kubelet/pods/90d8527c-7221-45b1-b2c5-9397fa8db4e3/volumes/kubernetes.io~projected/kube-api-access-8k8zx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9309b9f0-2a67-4c14-8af9-8759348ae5d5/volumes/kubernetes.io~projected/kube-api-access-wtc5z:{mountpoint:/var/lib/kubelet/pods/9309b9f0-2a67-4c14-8af9-8759348ae5d5/volumes/kubernetes.io~projected/kube-api-access-wtc5z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volumes/kubernetes.io~projected/kube-api-access-vf9vw:{mountpoint:/var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volumes/kubernetes.io~projected/kube-api-access-vf9vw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7f898aa-1924-459a-af1c-2242ac527e2a/volumes/kubernetes.io~projected/kube-api-access-q6w68:{mountpoint:/var/lib/kubelet/pods/d7f898aa-1924-459a-af1c-2242ac527e2a/volumes/kubernetes.io~projected/kube-api-access-q6w68 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~projected/kube-api-access-lfqnq:{mountpoint:/var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~projected/kube-api-access-lfqnq major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2595c92699d411fc002058c20a1e17469f8b78d6a79569c6d1e9550b5e45d06e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9cb44698a49a505e1aef7971cb51f1fe47aecc3982a450e28b001875ab13717b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/351865ea5dd68059ed2660f7a388dd3dd6b3a0240e0aa9e81cc49b43b15f4d97/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/39012099d946cc03ef803c4ee3d02c73ba90abb313d07df9a09bfd22bff9ea0e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/e8c173a0f6d5a0f7d031fd208392d0f8fe0fd740303fd7212cd7c403e4a635dc/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/30e2a3674f1cb6deb0e0c4ac5b6bc08408a67cf151e4e4742a9944ffb7fe8c17/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/12902401c5c5f5cf460df2b304426f9e9733f77a6d8d0cfc54e289527410c9f7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e3c2e1a5b408100ad8eb13e72221b469914fae98289e6819e1b95f0719bc3af5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/05af26fc825707ddbc0c9683a09a33940dbf0c118675948c3bafd16cef1e4a7a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/1e76b08e87a0212ee7c7e6f47a8d3cca8b82c1777c3a41afccdf516b3eac6476/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1c915f0caa056f3fe187f416c91d79214eaf86d57609dbcf4e2e501db898dae1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/49587a34e4906c7448fb7e8cfb402e98be1f26e3b7979fccf6a1dbd9f474dbf1/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c90b0d4852265d6effafa704398df014fde052c897742e11e4fa12e7bd8b7915/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8ced90d9405cf53c67d75293ed7ee8a59c4ab8a55f097f643222607e69b8f4ff/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a6443799c81dce086cfd27d5a0d19e76e4fba05a620978d0fbcab9c55e312862/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8897e4bfcc39459cf9ccffd62dce51251ddfa43ba4f301519d284b34c24df4f7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/83011b18b6b9216d3f6934ced4ed6d9261474bf0c7d9b5566c0da00e3bf0b871/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/38541d487d3935ef1b8f2c90f5bee57f072a63ad94a8cc67c47c4da235a8e5d3/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/4d496398df009ed900f169d1b701f0babed0b79235ce53de938f8a3dce473f32/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0df859ecd108e4465af0c8ad71b90821d297d1bbc8306011f2e83d5827d3a681/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bc5d022220397b7882e0a42d76be81c02474631997395c51ae740b4af2e0e618/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/51b6588d0a0aba1e307a4572b8ee32a72b2a7d82667264e6f93433749ecdeab0/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/57cb7e923701e3832159dc4011d7ebe1f3194b512c8b556c48c52a946964e611/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/27313aa01d8dc9091640df3ed8a812cada3c1b6993d9c5daabab455fdc3f0d0c/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4ce59dafbbb91ceb102ac618c29baf1741ddbb8e5929a39f37dbc7ac872ae858/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/49d7e1ce79c861f6d0594d6a5aa3bacc40c4aa1afc5e03434c0ae7530b001a8c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f24610ad5b28f6051672cfe119898742b2b4b07cabeb07217f0ff3f465252610/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/5577188470fb85cf8e45a1b978062161dccf3ca40301a44dca9d4210ad15d75c/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/dcb3277012e0cd212c05160927f63787cae71ad8c912f0adb970b82085cb82bf/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/dd83dd9256701543ca595be87fb688b4dfc09336e9535e4cc8f932c091d7dfa0/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/82948c31b44474706c058c399c0616aff006d0d8f7bb50cd7baccfa271ec34b5/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c9d54e79dc0ec677514ec9146b362b65dc3805488064babb7da7851b6609b712/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/9d245dd7d2294cbf6246712dedbdc0fbbb45accdff9d205feeb3bf2e96d10666/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/2accbe67d8d845d9080d17796d466ea1c1d73d26504c284e4ff88c6308921ed2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a130d117079eb476fa3ba6594565d3b6f48dcae58eae948965630a83d71298f6/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.376683 4314 manager.go:217] Machine: {Timestamp:2026-11-03 19:31:32.37623108 +0000 UTC m=+0.081070411 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100193 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e69c77088d9370497d5b1f162206 SystemUUID:ec29e69c-7708-8d93-7049-7d5b1f162206 BootID:c19d015b-d39c-4777-ab56-b9c31eb6a31a Filesystems:[{Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90d8527c-7221-45b1-b2c5-9397fa8db4e3/volumes/kubernetes.io~projected/kube-api-access-8k8zx DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volumes/kubernetes.io~projected/kube-api-access-vf9vw DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bf911cd-cd3c-4990-81ce-80d9ef41f214/volumes/kubernetes.io~projected/kube-api-access-t5q4r DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a69fa057d93699cf33a3f8219fd1adb86b4d4c02f763e9532e1f4c6137ae1e2e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83c3e661343f8afd2c02edc813790851a08b88c0f168ba1f7b251012ed1bdbc8/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~projected/kube-api-access-lfqnq DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cbe6aa8bcb1b8a94796ba0f2d28fc89e35cbd8e5f0bcc75d46dde2430abc478/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e17fbd17-d288-4823-bad6-cdd6d37e5c15/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9309b9f0-2a67-4c14-8af9-8759348ae5d5/volumes/kubernetes.io~projected/kube-api-access-wtc5z DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9aabebd259389a1a32ec28a8ebe43dd4ee6f51d0a1c4dd24de6199f9e7dabef7/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/656bc97e-5a5b-4334-8369-bc842712cf02/volumes/kubernetes.io~projected/kube-api-access-x74s2 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0225e321-66c4-4269-81d4-051793673f4d/volumes/kubernetes.io~projected/kube-api-access-fw6qm DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4278cb-b8ca-421a-804b-dc7c728ba346/volumes/kubernetes.io~projected/kube-api-access-d2zkv DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/957407ed50e65883014afa775cc085e9500e73441a343a28f2228df26942aaac/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d56c5c2-1d76-490b-b853-3e8d15662fa5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b6db40-add7-4eca-9480-e174e021fdc7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b71a2dfa0b931fa39b5ca0d882ef0b1da3a7a5d7c6e71ebd59a029a242edc1d0/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebf57f6c0b543b94e8a7226744d3ac9a460f414a560a799b9f092954c3975b05/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3b8a09106b087bfa2f77220293563f980f76ac6e654187168cc595690bb42a5/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9e625b9dd71b93087b9217529781377464a33122fe62cd15c87d02677fdebd9/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e10db87b493b86eb13afb4b3cf43392f57e8b52fb39fee404bdab1f2265aaee4/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d7f898aa-1924-459a-af1c-2242ac527e2a/volumes/kubernetes.io~projected/kube-api-access-q6w68 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fbf00d3-e30f-4399-b770-6545f69f15ce/volumes/kubernetes.io~projected/kube-api-access-nb56k DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:b71a2dfa0b931fa MacAddress:ee:48:cd:eb:4f:3e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fe:a5:b6:f1 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:e3:f2:2c:ea:0e Speed:0 Mtu:8901} {Name:e10db87b493b86e MacAddress:6a:81:27:f0:3d:1f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fe:a5:b6:f1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:66:02:45:a2:d3:c2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:1d:dc:62:a4:71 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.376887 4314 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.376926 4314 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377154 4314 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377172 4314 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-204.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377303 4314 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377310 4314 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377320 4314 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377340 4314 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377412 4314 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377551 4314 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377664 4314 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377696 4314 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:31:32.382904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377708 4314 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377718 4314 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377726 4314 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.377734 4314 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.379062 4314 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.379079 4314 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.380003 4314 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.381615 4314 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.381624 4314 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.381712 4314 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.381925 4314 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.381953 4314 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382037 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382050 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382059 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382067 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382072 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382078 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382083 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382089 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382096 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382101 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382114 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382129 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382158 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:31:32.383460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.382168 4314 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:31:32.385441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.385427 4314 server.go:1263] "Started kubelet" Nov 03 19:31:32.385533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.385505 4314 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:31:32.385596 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.385537 4314 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:31:32.385644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.385621 4314 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:31:32.385998 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.385977 4314 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:31:32.386020 ip-10-0-133-204 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:31:32.386947 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.386843 4314 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:31:32.387762 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.387694 4314 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:31:32.388384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.388364 4314 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:31:32.388761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.388719 4314 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:54 +0000 UTC" deadline="2028-04-20 22:13:54.110646701 +0000 UTC" Nov 03 19:31:32.388761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.388760 4314 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12818h42m21.721889159s" Nov 03 19:31:32.388902 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:32.388790 4314 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-204.us-east-2.compute.internal\" not found" Nov 03 19:31:32.388902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.388831 4314 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:31:32.388902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.388840 4314 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:31:32.389092 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.388959 4314 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:31:32.389858 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.389836 4314 factory.go:55] Registering systemd factory Nov 03 19:31:32.389858 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.389858 4314 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:31:32.390249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.390181 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf911cd-cd3c-4990-81ce-80d9ef41f214" volumeName="kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 19:31:32.390359 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.390254 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b6db40-add7-4eca-9480-e174e021fdc7" volumeName="kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-script-lib" seLinuxMountContext="" Nov 03 19:31:32.393431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393382 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e17fbd17-d288-4823-bad6-cdd6d37e5c15" volumeName="kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-tuned" seLinuxMountContext="" Nov 03 19:31:32.393531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393450 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e17fbd17-d288-4823-bad6-cdd6d37e5c15" volumeName="kubernetes.io/projected/e17fbd17-d288-4823-bad6-cdd6d37e5c15-kube-api-access-lfqnq" seLinuxMountContext="" Nov 03 19:31:32.393531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393470 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0225e321-66c4-4269-81d4-051793673f4d" volumeName="kubernetes.io/empty-dir/0225e321-66c4-4269-81d4-051793673f4d-tmp-dir" seLinuxMountContext="" Nov 03 19:31:32.393531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393494 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="656bc97e-5a5b-4334-8369-bc842712cf02" volumeName="kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2" seLinuxMountContext="" Nov 03 19:31:32.393531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393510 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bac36d0-ca54-4a9c-83cd-c58ba9f58c37" volumeName="kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393547 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf911cd-cd3c-4990-81ce-80d9ef41f214" volumeName="kubernetes.io/projected/8bf911cd-cd3c-4990-81ce-80d9ef41f214-kube-api-access-t5q4r" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393576 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b6db40-add7-4eca-9480-e174e021fdc7" volumeName="kubernetes.io/secret/c8b6db40-add7-4eca-9480-e174e021fdc7-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393616 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7f898aa-1924-459a-af1c-2242ac527e2a" volumeName="kubernetes.io/projected/d7f898aa-1924-459a-af1c-2242ac527e2a-kube-api-access-q6w68" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393628 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d56c5c2-1d76-490b-b853-3e8d15662fa5" volumeName="kubernetes.io/configmap/2d56c5c2-1d76-490b-b853-3e8d15662fa5-konnectivity-ca" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393640 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbf00d3-e30f-4399-b770-6545f69f15ce" volumeName="kubernetes.io/projected/3fbf00d3-e30f-4399-b770-6545f69f15ce-kube-api-access-nb56k" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393658 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf911cd-cd3c-4990-81ce-80d9ef41f214" volumeName="kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-binary-copy" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393677 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b6db40-add7-4eca-9480-e174e021fdc7" volumeName="kubernetes.io/projected/c8b6db40-add7-4eca-9480-e174e021fdc7-kube-api-access-vf9vw" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393700 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e17fbd17-d288-4823-bad6-cdd6d37e5c15" volumeName="kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-tmp" seLinuxMountContext="" Nov 03 19:31:32.393735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393720 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4278cb-b8ca-421a-804b-dc7c728ba346" volumeName="kubernetes.io/configmap/4e4278cb-b8ca-421a-804b-dc7c728ba346-iptables-alerter-script" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393737 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bf911cd-cd3c-4990-81ce-80d9ef41f214" volumeName="kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393755 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90d8527c-7221-45b1-b2c5-9397fa8db4e3" volumeName="kubernetes.io/projected/90d8527c-7221-45b1-b2c5-9397fa8db4e3-kube-api-access-8k8zx" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393767 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9309b9f0-2a67-4c14-8af9-8759348ae5d5" volumeName="kubernetes.io/projected/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kube-api-access-wtc5z" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393808 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b6db40-add7-4eca-9480-e174e021fdc7" volumeName="kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-config" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393821 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b6db40-add7-4eca-9480-e174e021fdc7" volumeName="kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-env-overrides" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393845 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7f898aa-1924-459a-af1c-2242ac527e2a" volumeName="kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-cni-binary-copy" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393865 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7f898aa-1924-459a-af1c-2242ac527e2a" volumeName="kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-daemon-config" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393873 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0225e321-66c4-4269-81d4-051793673f4d" volumeName="kubernetes.io/projected/0225e321-66c4-4269-81d4-051793673f4d-kube-api-access-fw6qm" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393883 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d56c5c2-1d76-490b-b853-3e8d15662fa5" volumeName="kubernetes.io/secret/2d56c5c2-1d76-490b-b853-3e8d15662fa5-agent-certs" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393907 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbf00d3-e30f-4399-b770-6545f69f15ce" volumeName="kubernetes.io/configmap/3fbf00d3-e30f-4399-b770-6545f69f15ce-serviceca" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393917 4314 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4278cb-b8ca-421a-804b-dc7c728ba346" volumeName="kubernetes.io/projected/4e4278cb-b8ca-421a-804b-dc7c728ba346-kube-api-access-d2zkv" seLinuxMountContext="" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393931 4314 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:31:32.394129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.393936 4314 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:31:32.394812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.394793 4314 factory.go:153] Registering CRI-O factory Nov 03 19:31:32.394886 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.394815 4314 factory.go:223] Registration of the crio container factory successfully Nov 03 19:31:32.394886 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.394883 4314 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:31:32.394993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.394899 4314 factory.go:103] Registering Raw factory Nov 03 19:31:32.394993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.394910 4314 manager.go:1199] Started watching for new ooms in manager Nov 03 19:31:32.395184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.395171 4314 manager.go:319] Starting recovery of all containers Nov 03 19:31:32.398592 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.397886 4314 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:31:32.399771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.399680 4314 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 19:31:32.399933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.399911 4314 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:31:32.400356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.400322 4314 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:31:32.412180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.412161 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:32.413554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.413537 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:32.418519 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.418471 4314 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:31:32.419812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.419790 4314 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:31:32.419812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.419814 4314 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:31:32.419955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.419835 4314 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:31:32.419955 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:32.419876 4314 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:31:32.423732 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.423710 4314 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:31:32.427401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.427376 4314 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="6e5a017536a472d02d7817180f8ae2e33c03293bac513ae9e239a5426bf2c35e" exitCode=0 Nov 03 19:31:32.427401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.427399 4314 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="59cf718db877ac07bd14fd4f81cbad79976f6966b0675886be5a5bcfc2092e59" exitCode=0 Nov 03 19:31:32.427492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.427407 4314 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="de46499ea63a40f0b7f94a761163a5e86e98111a8cea40e72d5d85140b47c56d" exitCode=0 Nov 03 19:31:32.427492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.427414 4314 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="7876dc84bf2a66a9b2aff2a4a8b63a302a85ef603598baf021dc130da50fa65a" exitCode=0 Nov 03 19:31:32.427492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.427421 4314 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="dfff7bfdc83388e91e370d3d88e8f20e98dd1e5d05d88ea50fc854989b84bd79" exitCode=0 Nov 03 19:31:32.427492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.427427 4314 generic.go:356] "Generic (PLEG): container finished" podID="8bf911cd-cd3c-4990-81ce-80d9ef41f214" containerID="d940c9652647e7e79592dfd6e720ebd0aab26c44bf415521ade1438f6a2105c2" exitCode=0 Nov 03 19:31:32.435033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.435016 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:31:32.435309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.435286 4314 generic.go:356] "Generic (PLEG): container finished" podID="c8b6db40-add7-4eca-9480-e174e021fdc7" containerID="6118976b097e9e300e55553fc2dee54187c5da1bf122c99c828d6087e62e71ee" exitCode=1 Nov 03 19:31:32.438430 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.438410 4314 generic.go:356] "Generic (PLEG): container finished" podID="300bf0c34ec8efd4851a1973652985e3" containerID="dae363d801e26fa6dbd3eb3a6dffa7851f0f9d7f4fc415d5b0fbe8110e30c218" exitCode=0 Nov 03 19:31:32.455295 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.455277 4314 manager.go:324] Recovery completed Nov 03 19:31:32.467089 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467071 4314 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:31:32.467089 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467083 4314 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:31:32.467182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467098 4314 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:31:32.467211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467197 4314 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 19:31:32.467249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467207 4314 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 19:31:32.467249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467222 4314 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 19:31:32.467249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467227 4314 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 19:31:32.467249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467231 4314 policy_none.go:50] "Start" Nov 03 19:31:32.467249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467239 4314 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:31:32.467249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467247 4314 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:31:32.467395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467357 4314 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 19:31:32.467395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467363 4314 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 19:31:32.467395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.467370 4314 policy_none.go:44] "Start" Nov 03 19:31:32.472260 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.472235 4314 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:31:32.474884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.474870 4314 manager.go:341] "Starting Device Plugin manager" Nov 03 19:31:32.475530 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:32.474893 4314 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:31:32.475530 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.475528 4314 server.go:87] "Starting device plugin registration server" Nov 03 19:31:32.475793 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.475780 4314 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:31:32.475853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.475791 4314 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:31:32.475924 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.475878 4314 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:31:32.476015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.475975 4314 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:31:32.476074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.476028 4314 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:31:32.476074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.476037 4314 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:31:32.476175 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.476096 4314 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:31:32.476015342Z","UUID":"ae5bb6c1-4733-4ce7-ada9-f4a328a29292","Handler":null,"Name":"","Endpoint":""} Nov 03 19:31:32.477257 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:32.477238 4314 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:31:32.477383 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.477364 4314 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:31:32.477476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.477389 4314 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:31:32.520726 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.520691 4314 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal"] Nov 03 19:31:32.521194 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.521159 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"70af56b747e8dbbb62b5b41a5fc7a4ec","Type":"ContainerStarted","Data":"97480ce46a668e43467d52aef3b48208b56742283e9d3134eba41c60367a7040"} Nov 03 19:31:32.521257 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.521206 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"70af56b747e8dbbb62b5b41a5fc7a4ec","Type":"ContainerStarted","Data":"83258c3347072e0e18e5aeee8828c21f7266574c8277d8176554517e650290dc"} Nov 03 19:31:32.521257 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.521228 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"300bf0c34ec8efd4851a1973652985e3","Type":"ContainerStarted","Data":"0c2e2cad3c00d597419183037a3c0bd3a380eb7df2f661df23c0212e8bca6c03"} Nov 03 19:31:32.521257 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.521251 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"300bf0c34ec8efd4851a1973652985e3","Type":"ContainerDied","Data":"dae363d801e26fa6dbd3eb3a6dffa7851f0f9d7f4fc415d5b0fbe8110e30c218"} Nov 03 19:31:32.521375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.521260 4314 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.521375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.521277 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" event={"ID":"300bf0c34ec8efd4851a1973652985e3","Type":"ContainerStarted","Data":"7b560c1a3b8710aeed51c592a9e90819f389aea8db8a03a4cc73bffcf875d3c9"} Nov 03 19:31:32.521434 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.521409 4314 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.530577 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.530560 4314 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:31:32.530655 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:32.530609 4314 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.530922 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.530909 4314 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:31:32.530955 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:32.530942 4314 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.576043 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.576026 4314 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:31:32.577153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.577134 4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:31:32.577234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.577161 4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:31:32.577234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.577171 4314 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-204.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:31:32.577234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.577204 4314 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.589539 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.589518 4314 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.589639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.589568 4314 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.601566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.601542 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.601661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.601572 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.601661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.601611 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70af56b747e8dbbb62b5b41a5fc7a4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"70af56b747e8dbbb62b5b41a5fc7a4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.702594 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.702564 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70af56b747e8dbbb62b5b41a5fc7a4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"70af56b747e8dbbb62b5b41a5fc7a4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.702677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.702617 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.702677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.702643 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.702677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.702666 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70af56b747e8dbbb62b5b41a5fc7a4ec-config\") pod \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"70af56b747e8dbbb62b5b41a5fc7a4ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.702797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.702696 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:32.702797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:32.702730 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/300bf0c34ec8efd4851a1973652985e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" (UID: \"300bf0c34ec8efd4851a1973652985e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:33.378720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.378692 4314 apiserver.go:52] "Watching apiserver" Nov 03 19:31:33.387715 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.387694 4314 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:31:33.388104 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.388079 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-vhgfr","openshift-dns/node-resolver-ns5zn","openshift-multus/network-metrics-daemon-bbtrq","openshift-network-operator/iptables-alerter-629fw","kube-system/global-pull-secret-syncer-8z8g9","kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-jf9qv","openshift-multus/multus-k6fjx","openshift-network-diagnostics/network-check-target-pp285","openshift-image-registry/node-ca-zj9jh","openshift-ovn-kubernetes/ovnkube-node-xss4s","kube-system/konnectivity-agent-8cmvr"] Nov 03 19:31:33.389265 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.389240 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:33.392349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.392327 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:31:33.393940 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.393922 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:31:33.394019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.393948 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:31:33.394019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.393952 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.394118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.394081 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.394324 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.394302 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.394390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.394336 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.394390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.394346 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bh6bb\"" Nov 03 19:31:33.394390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.394336 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:31:33.394390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.394378 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.394508 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.394448 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-glk7q\"" Nov 03 19:31:33.395523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.395496 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:31:33.395620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.395564 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tdq2h\"" Nov 03 19:31:33.395736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.395718 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.395803 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.395731 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:31:33.396158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396136 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.396242 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396225 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6x7r\"" Nov 03 19:31:33.396297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396262 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:31:33.396457 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396438 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:31:33.396509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396482 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:31:33.396691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396673 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:31:33.396900 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396882 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.397000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.396986 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.397058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397011 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.397105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397057 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9h8df\"" Nov 03 19:31:33.397208 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397189 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.397325 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397309 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kqxn2\"" Nov 03 19:31:33.397418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397397 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:31:33.397482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397350 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khl78\"" Nov 03 19:31:33.397536 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397511 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.397536 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397527 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:31:33.397843 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397332 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.397920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397904 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:31:33.397981 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397926 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wllhj\"" Nov 03 19:31:33.397981 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397944 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6p8wh\"" Nov 03 19:31:33.397981 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397851 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77jlf\"" Nov 03 19:31:33.398114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.398012 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.398114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397859 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgldx\"" Nov 03 19:31:33.398114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397372 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:31:33.398114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.397945 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:31:33.398282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.398116 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pq7nf\"" Nov 03 19:31:33.398282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.398260 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:31:33.398435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.398417 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:31:33.405550 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405529 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.405639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405559 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.405639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405596 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-kubelet\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.405710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405663 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-slash\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.405710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405698 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-systemd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.405790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405721 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-var-lib-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.405790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405749 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.405790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405770 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-systemd-units\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.405790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405788 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-node-log\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.405939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405840 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-netd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.405939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405878 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e4278cb-b8ca-421a-804b-dc7c728ba346-host-slash\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:31:33.405939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405906 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zkv\" (UniqueName: \"kubernetes.io/projected/4e4278cb-b8ca-421a-804b-dc7c728ba346-kube-api-access-d2zkv\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:31:33.405939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405921 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.406087 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.405964 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-netns\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406087 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406002 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-ovn\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406087 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406026 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:33.406087 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406059 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-log-socket\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406221 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406103 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e4278cb-b8ca-421a-804b-dc7c728ba346-iptables-alerter-script\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:31:33.406221 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406145 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-env-overrides\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406285 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406346 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406312 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-env-overrides\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406373 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406348 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8zx\" (UniqueName: \"kubernetes.io/projected/90d8527c-7221-45b1-b2c5-9397fa8db4e3-kube-api-access-8k8zx\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:33.406402 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406379 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtc5z\" (UniqueName: \"kubernetes.io/projected/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kube-api-access-wtc5z\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.406483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406466 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406476 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e4278cb-b8ca-421a-804b-dc7c728ba346-iptables-alerter-script\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:31:33.406542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406498 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-config\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406537 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406566 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b6db40-add7-4eca-9480-e174e021fdc7-ovn-node-metrics-cert\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406618 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9vw\" (UniqueName: \"kubernetes.io/projected/c8b6db40-add7-4eca-9480-e174e021fdc7-kube-api-access-vf9vw\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406639 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-device-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.406696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406662 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.406757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406693 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-config\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406701 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-etc-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406749 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-bin\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406837 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406777 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-script-lib\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.406969 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.406952 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8b6db40-add7-4eca-9480-e174e021fdc7-ovnkube-script-lib\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.407046 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.407034 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8b6db40-add7-4eca-9480-e174e021fdc7-ovn-node-metrics-cert\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.412635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.412618 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtc5z\" (UniqueName: \"kubernetes.io/projected/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kube-api-access-wtc5z\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.412709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.412670 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8zx\" (UniqueName: \"kubernetes.io/projected/90d8527c-7221-45b1-b2c5-9397fa8db4e3-kube-api-access-8k8zx\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:33.413140 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.413125 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9vw\" (UniqueName: \"kubernetes.io/projected/c8b6db40-add7-4eca-9480-e174e021fdc7-kube-api-access-vf9vw\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.413291 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.413271 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zkv\" (UniqueName: \"kubernetes.io/projected/4e4278cb-b8ca-421a-804b-dc7c728ba346-kube-api-access-d2zkv\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:31:33.452175 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.452156 4314 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:33.452325 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.452311 4314 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:33.463690 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.463669 4314 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:31:33.463770 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:33.463709 4314 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:33.463770 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.463669 4314 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:31:33.463831 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:33.463786 4314 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal" Nov 03 19:31:33.489803 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.489775 4314 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:31:33.507117 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507093 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-var-lib-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507123 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-k8s-cni-cncf-io\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.507203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507139 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.507203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507157 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-systemd-units\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507209 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-node-log\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507225 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-var-lib-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507234 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-netd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507255 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-systemd-units\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507262 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507304 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-systemd\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507311 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-netd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507329 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-sys\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507364 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-netns\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507382 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-ovn\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507428 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-netns\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507440 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-node-log\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507450 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6w68\" (UniqueName: \"kubernetes.io/projected/d7f898aa-1924-459a-af1c-2242ac527e2a-kube-api-access-q6w68\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507485 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqnq\" (UniqueName: \"kubernetes.io/projected/e17fbd17-d288-4823-bad6-cdd6d37e5c15-kube-api-access-lfqnq\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.507509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507511 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-binary-copy\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507526 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-log-socket\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507560 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-etc-kubernetes\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507570 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-ovn\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507598 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-log-socket\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507606 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d56c5c2-1d76-490b-b853-3e8d15662fa5-konnectivity-ca\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507625 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507660 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-run\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507681 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-netns\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507699 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb56k\" (UniqueName: \"kubernetes.io/projected/3fbf00d3-e30f-4399-b770-6545f69f15ce-kube-api-access-nb56k\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507713 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-tuned\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507754 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507839 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507855 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507859 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d56c5c2-1d76-490b-b853-3e8d15662fa5-agent-certs\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507876 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-tuned\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.507901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507896 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d56c5c2-1d76-490b-b853-3e8d15662fa5-konnectivity-ca\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507910 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-system-cni-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507950 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-socket-dir-parent\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.507978 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-bin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508014 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-kubelet\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508040 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3fbf00d3-e30f-4399-b770-6545f69f15ce-serviceca\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508061 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508096 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d56c5c2-1d76-490b-b853-3e8d15662fa5-agent-certs\") pod \"konnectivity-agent-8cmvr\" (UID: \"2d56c5c2-1d76-490b-b853-3e8d15662fa5\") " pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508162 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-host\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508185 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-device-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508205 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bf911cd-cd3c-4990-81ce-80d9ef41f214-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508222 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0225e321-66c4-4269-81d4-051793673f4d-tmp-dir\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508261 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-device-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508250 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-multus\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508310 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-daemon-config\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508334 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0225e321-66c4-4269-81d4-051793673f4d-tmp-dir\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:31:33.508388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508372 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508393 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-slash\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508419 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6qm\" (UniqueName: \"kubernetes.io/projected/0225e321-66c4-4269-81d4-051793673f4d-kube-api-access-fw6qm\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508445 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508445 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-slash\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508504 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-cnibin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508543 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508570 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-var-lib-kubelet\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508610 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508657 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508663 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e4278cb-b8ca-421a-804b-dc7c728ba346-host-slash\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508692 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-dbus\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508706 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3fbf00d3-e30f-4399-b770-6545f69f15ce-serviceca\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508724 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-daemon-config\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508739 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e4278cb-b8ca-421a-804b-dc7c728ba346-host-slash\") pod \"iptables-alerter-629fw\" (UID: \"4e4278cb-b8ca-421a-804b-dc7c728ba346\") " pod="openshift-network-operator/iptables-alerter-629fw" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508778 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-conf\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508797 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508813 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508828 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-system-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508843 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-cni-binary-copy\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508846 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.508929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508861 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cnibin\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.508966 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509065 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysconfig\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509094 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f898aa-1924-459a-af1c-2242ac527e2a-cni-binary-copy\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509131 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-kubelet-config\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509147 4314 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509231 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-kubernetes\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509248 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-tmp\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509263 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509362 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509395 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0225e321-66c4-4269-81d4-051793673f4d-hosts-file\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509423 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e17fbd17-d288-4823-bad6-cdd6d37e5c15-tmp\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509430 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-conf-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509475 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-lib-modules\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509524 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509571 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-run-ovn-kubernetes\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509576 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-multus-certs\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509628 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fbf00d3-e30f-4399-b770-6545f69f15ce-host\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509654 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5q4r\" (UniqueName: \"kubernetes.io/projected/8bf911cd-cd3c-4990-81ce-80d9ef41f214-kube-api-access-t5q4r\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509678 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509692 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-etc-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509715 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-bin\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.509741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509743 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509767 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509775 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509781 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-etc-openvswitch\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509792 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-modprobe-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509843 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-hostroot\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509868 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509898 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-kubelet\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509944 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-kubelet\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509947 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9309b9f0-2a67-4c14-8af9-8759348ae5d5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2xh5\" (UID: \"9309b9f0-2a67-4c14-8af9-8759348ae5d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2xh5" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.509975 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-systemd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.510030 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-run-systemd\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.510041 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8b6db40-add7-4eca-9480-e174e021fdc7-host-cni-bin\") pod \"ovnkube-node-xss4s\" (UID: \"c8b6db40-add7-4eca-9480-e174e021fdc7\") " pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:33.510847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.510098 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-original-pull-secret\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.511257 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.511234 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d8527c-7221-45b1-b2c5-9397fa8db4e3-metrics-certs\") pod \"network-metrics-daemon-bbtrq\" (UID: \"90d8527c-7221-45b1-b2c5-9397fa8db4e3\") " pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:33.513595 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.513564 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqnq\" (UniqueName: \"kubernetes.io/projected/e17fbd17-d288-4823-bad6-cdd6d37e5c15-kube-api-access-lfqnq\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.513702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.513643 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6w68\" (UniqueName: \"kubernetes.io/projected/d7f898aa-1924-459a-af1c-2242ac527e2a-kube-api-access-q6w68\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.513910 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.513891 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb56k\" (UniqueName: \"kubernetes.io/projected/3fbf00d3-e30f-4399-b770-6545f69f15ce-kube-api-access-nb56k\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:31:33.514327 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.514291 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6qm\" (UniqueName: \"kubernetes.io/projected/0225e321-66c4-4269-81d4-051793673f4d-kube-api-access-fw6qm\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:31:33.515363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.515342 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5q4r\" (UniqueName: \"kubernetes.io/projected/8bf911cd-cd3c-4990-81ce-80d9ef41f214-kube-api-access-t5q4r\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.515780 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.515761 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74s2\" (UniqueName: \"kubernetes.io/projected/656bc97e-5a5b-4334-8369-bc842712cf02-kube-api-access-x74s2\") pod \"network-check-target-pp285\" (UID: \"656bc97e-5a5b-4334-8369-bc842712cf02\") " pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:33.610646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610622 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-modprobe-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.610764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610655 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-hostroot\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.610764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610681 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-k8s-cni-cncf-io\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.610764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610718 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-systemd\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.610764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610738 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-sys\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.610764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610745 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-hostroot\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610785 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-etc-kubernetes\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610799 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-k8s-cni-cncf-io\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610803 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-modprobe-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610822 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-run\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610834 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-etc-kubernetes\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610823 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-systemd\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610847 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-netns\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610857 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-run\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610855 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-sys\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610878 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610904 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-system-cni-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610904 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-netns\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611004 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-socket-dir-parent\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611012 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-system-cni-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.610993 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611033 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-bin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611063 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-kubelet\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611079 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-host\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611039 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-socket-dir-parent\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611071 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-bin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611099 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-multus\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611125 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-kubelet\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611128 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-var-lib-cni-multus\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611132 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-cnibin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611131 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-host\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611175 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611222 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-var-lib-kubelet\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611231 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-cnibin\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611255 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-dbus\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611294 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-var-lib-kubelet\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611316 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-conf\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611329 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-d\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611314 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-dbus\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611338 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-system-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611363 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-system-cni-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611364 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cnibin\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611392 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611412 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-cnibin\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611419 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysconfig\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611435 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-kubelet-config\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611448 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-kubernetes\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611447 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bf911cd-cd3c-4990-81ce-80d9ef41f214-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jf9qv\" (UID: \"8bf911cd-cd3c-4990-81ce-80d9ef41f214\") " pod="openshift-multus/multus-additional-cni-plugins-jf9qv" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611455 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysctl-conf\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611465 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0225e321-66c4-4269-81d4-051793673f4d-hosts-file\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611476 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-sysconfig\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611479 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-conf-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611498 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7bac36d0-ca54-4a9c-83cd-c58ba9f58c37-kubelet-config\") pod \"global-pull-secret-syncer-8z8g9\" (UID: \"7bac36d0-ca54-4a9c-83cd-c58ba9f58c37\") " pod="kube-system/global-pull-secret-syncer-8z8g9" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611501 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-multus-conf-dir\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611512 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-etc-kubernetes\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611514 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-lib-modules\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611508 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0225e321-66c4-4269-81d4-051793673f4d-hosts-file\") pod \"node-resolver-ns5zn\" (UID: \"0225e321-66c4-4269-81d4-051793673f4d\") " pod="openshift-dns/node-resolver-ns5zn" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611574 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-multus-certs\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611611 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e17fbd17-d288-4823-bad6-cdd6d37e5c15-lib-modules\") pod \"tuned-vhgfr\" (UID: \"e17fbd17-d288-4823-bad6-cdd6d37e5c15\") " pod="openshift-cluster-node-tuning-operator/tuned-vhgfr" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611631 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fbf00d3-e30f-4399-b770-6545f69f15ce-host\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611642 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7f898aa-1924-459a-af1c-2242ac527e2a-host-run-multus-certs\") pod \"multus-k6fjx\" (UID: \"d7f898aa-1924-459a-af1c-2242ac527e2a\") " pod="openshift-multus/multus-k6fjx" Nov 03 19:31:33.611723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.611663 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3fbf00d3-e30f-4399-b770-6545f69f15ce-host\") pod \"node-ca-zj9jh\" (UID: \"3fbf00d3-e30f-4399-b770-6545f69f15ce\") " pod="openshift-image-registry/node-ca-zj9jh" Nov 03 19:31:33.689953 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.689899 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bbtrq" Nov 03 19:31:33.988025 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.987957 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bbtrq"] Nov 03 19:31:33.997623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:33.997601 4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:31:34.454910 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:34.454874 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bbtrq" event={"ID":"90d8527c-7221-45b1-b2c5-9397fa8db4e3","Type":"ContainerStarted","Data":"1aaa72390dcbe41c80d2d5e88a26c6a86419359144e8a3ba73e0a7dfe1ac9f50"} Nov 03 19:31:35.012231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:35.012203 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8cmvr" Nov 03 19:31:36.461423 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:36.461387 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bbtrq" event={"ID":"90d8527c-7221-45b1-b2c5-9397fa8db4e3","Type":"ContainerStarted","Data":"c4f8f39e2bb7b6d210cca4b4065084d3c8e7c4fe0da3ba2cee9c5b8ed917ed8c"} Nov 03 19:31:36.461895 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:36.461439 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bbtrq" event={"ID":"90d8527c-7221-45b1-b2c5-9397fa8db4e3","Type":"ContainerStarted","Data":"ba838aaf84fb03baa274f5ab87900bf5779563054edb8ea3d78f8b76571d8702"} Nov 03 19:31:36.957775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:36.957744 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:31:39.802213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.802181 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rlqmb"] Nov 03 19:31:39.804022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.804005 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.816019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.815998 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt"] Nov 03 19:31:39.817475 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.817458 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6gx66"] Nov 03 19:31:39.817610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.817590 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" Nov 03 19:31:39.819429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.819412 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:39.821822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.821242 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:31:39.821822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.821443 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:31:39.821822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.821535 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brhwf\"" Nov 03 19:31:39.821822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.821763 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlqmb"] Nov 03 19:31:39.830420 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.830402 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:31:39.830508 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.830444 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:31:39.830508 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.830498 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:31:39.832712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.832691 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt"] Nov 03 19:31:39.833128 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.833097 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6gx66"] Nov 03 19:31:39.842570 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.842527 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e287d664-857c-4fb1-beb0-f248ef548609-tmp-dir\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.842570 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.842566 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mg4\" (UniqueName: \"kubernetes.io/projected/e287d664-857c-4fb1-beb0-f248ef548609-kube-api-access-c4mg4\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.842782 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.842647 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e287d664-857c-4fb1-beb0-f248ef548609-config-volume\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.842782 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.842701 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e287d664-857c-4fb1-beb0-f248ef548609-metrics-tls\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.842887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.842779 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4ab98eaf-1317-4f84-aab7-b3a89831ea07-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cjcmt\" (UID: \"4ab98eaf-1317-4f84-aab7-b3a89831ea07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" Nov 03 19:31:39.854091 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.854067 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c6c8z"] Nov 03 19:31:39.855697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.855682 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:39.868471 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.868446 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:31:39.868627 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.868573 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:31:39.869656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.869632 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6c8z"] Nov 03 19:31:39.896772 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.896748 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:39.909866 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.909841 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57899686b7-qh8vj"] Nov 03 19:31:39.911443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.911427 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.915652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915628 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:31:39.915652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915628 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 19:31:39.915799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915663 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:31:39.915799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915751 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sh2j9\"" Nov 03 19:31:39.915799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915754 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rgxdj\"" Nov 03 19:31:39.915899 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915810 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nblsj\"" Nov 03 19:31:39.915899 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915830 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-skb8s\"" Nov 03 19:31:39.915957 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915898 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:31:39.915957 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.915904 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:31:39.924884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.924862 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:31:39.926371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.926353 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57899686b7-qh8vj"] Nov 03 19:31:39.943130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943108 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-registry-tls\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.943235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943138 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d834ce-9da4-4dfe-8433-89df929dd609-ca-trust-extracted\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.943235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943170 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e287d664-857c-4fb1-beb0-f248ef548609-config-volume\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.943235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943185 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e287d664-857c-4fb1-beb0-f248ef548609-metrics-tls\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.943371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943350 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62450e22-1ab9-4663-8960-cceab528bd29-data-volume\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:39.943464 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943435 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vp5h\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-kube-api-access-8vp5h\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.943532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943493 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4ab98eaf-1317-4f84-aab7-b3a89831ea07-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cjcmt\" (UID: \"4ab98eaf-1317-4f84-aab7-b3a89831ea07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" Nov 03 19:31:39.943532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943520 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d834ce-9da4-4dfe-8433-89df929dd609-image-registry-private-configuration\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.943664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943543 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-bound-sa-token\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.943664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943656 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc6vs\" (UniqueName: \"kubernetes.io/projected/62450e22-1ab9-4663-8960-cceab528bd29-kube-api-access-fc6vs\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:39.943778 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943735 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62450e22-1ab9-4663-8960-cceab528bd29-crio-socket\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:39.943778 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943770 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d834ce-9da4-4dfe-8433-89df929dd609-installation-pull-secrets\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.943875 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943804 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e287d664-857c-4fb1-beb0-f248ef548609-tmp-dir\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.943875 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943837 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e287d664-857c-4fb1-beb0-f248ef548609-config-volume\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.943875 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.943839 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mg4\" (UniqueName: \"kubernetes.io/projected/e287d664-857c-4fb1-beb0-f248ef548609-kube-api-access-c4mg4\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.944043 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.944019 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fd524a3-fa08-43aa-8db1-71bb34dd961a-cert\") pod \"ingress-canary-c6c8z\" (UID: \"9fd524a3-fa08-43aa-8db1-71bb34dd961a\") " pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:39.944136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.944086 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fpd\" (UniqueName: \"kubernetes.io/projected/9fd524a3-fa08-43aa-8db1-71bb34dd961a-kube-api-access-98fpd\") pod \"ingress-canary-c6c8z\" (UID: \"9fd524a3-fa08-43aa-8db1-71bb34dd961a\") " pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:39.944136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.944123 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62450e22-1ab9-4663-8960-cceab528bd29-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:39.944214 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.944162 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e287d664-857c-4fb1-beb0-f248ef548609-tmp-dir\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.944214 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.944189 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62450e22-1ab9-4663-8960-cceab528bd29-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:39.944285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.944213 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d834ce-9da4-4dfe-8433-89df929dd609-registry-certificates\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.944285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.944229 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d834ce-9da4-4dfe-8433-89df929dd609-trusted-ca\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:39.945554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.945537 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e287d664-857c-4fb1-beb0-f248ef548609-metrics-tls\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:39.945796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.945776 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4ab98eaf-1317-4f84-aab7-b3a89831ea07-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cjcmt\" (UID: \"4ab98eaf-1317-4f84-aab7-b3a89831ea07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" Nov 03 19:31:39.952318 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:39.952291 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mg4\" (UniqueName: \"kubernetes.io/projected/e287d664-857c-4fb1-beb0-f248ef548609-kube-api-access-c4mg4\") pod \"dns-default-rlqmb\" (UID: \"e287d664-857c-4fb1-beb0-f248ef548609\") " pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:40.015348 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.015319 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:31:40.045045 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045024 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d834ce-9da4-4dfe-8433-89df929dd609-installation-pull-secrets\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045059 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fd524a3-fa08-43aa-8db1-71bb34dd961a-cert\") pod \"ingress-canary-c6c8z\" (UID: \"9fd524a3-fa08-43aa-8db1-71bb34dd961a\") " pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:40.045157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045084 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fpd\" (UniqueName: \"kubernetes.io/projected/9fd524a3-fa08-43aa-8db1-71bb34dd961a-kube-api-access-98fpd\") pod \"ingress-canary-c6c8z\" (UID: \"9fd524a3-fa08-43aa-8db1-71bb34dd961a\") " pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:40.045157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045107 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62450e22-1ab9-4663-8960-cceab528bd29-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.045258 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045229 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62450e22-1ab9-4663-8960-cceab528bd29-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.045292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045261 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d834ce-9da4-4dfe-8433-89df929dd609-registry-certificates\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045278 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d834ce-9da4-4dfe-8433-89df929dd609-trusted-ca\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045373 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045295 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-registry-tls\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045412 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d834ce-9da4-4dfe-8433-89df929dd609-ca-trust-extracted\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045497 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045453 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62450e22-1ab9-4663-8960-cceab528bd29-data-volume\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.045497 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045478 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vp5h\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-kube-api-access-8vp5h\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045627 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045517 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d834ce-9da4-4dfe-8433-89df929dd609-image-registry-private-configuration\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045627 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045601 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-bound-sa-token\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.045742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045648 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc6vs\" (UniqueName: \"kubernetes.io/projected/62450e22-1ab9-4663-8960-cceab528bd29-kube-api-access-fc6vs\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.045742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045683 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62450e22-1ab9-4663-8960-cceab528bd29-crio-socket\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.045853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.045806 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62450e22-1ab9-4663-8960-cceab528bd29-crio-socket\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.046677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.046122 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62450e22-1ab9-4663-8960-cceab528bd29-data-volume\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.046677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.046174 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2d834ce-9da4-4dfe-8433-89df929dd609-ca-trust-extracted\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.046677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.046209 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2d834ce-9da4-4dfe-8433-89df929dd609-registry-certificates\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.046677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.046482 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2d834ce-9da4-4dfe-8433-89df929dd609-trusted-ca\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.046677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.046672 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62450e22-1ab9-4663-8960-cceab528bd29-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.047967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.047938 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-registry-tls\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.048063 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.048032 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62450e22-1ab9-4663-8960-cceab528bd29-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.048140 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.048082 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2d834ce-9da4-4dfe-8433-89df929dd609-installation-pull-secrets\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.048140 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.048127 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fd524a3-fa08-43aa-8db1-71bb34dd961a-cert\") pod \"ingress-canary-c6c8z\" (UID: \"9fd524a3-fa08-43aa-8db1-71bb34dd961a\") " pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:40.048568 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.048550 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2d834ce-9da4-4dfe-8433-89df929dd609-image-registry-private-configuration\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.056556 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.056500 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fpd\" (UniqueName: \"kubernetes.io/projected/9fd524a3-fa08-43aa-8db1-71bb34dd961a-kube-api-access-98fpd\") pod \"ingress-canary-c6c8z\" (UID: \"9fd524a3-fa08-43aa-8db1-71bb34dd961a\") " pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:40.056720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.056700 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-bound-sa-token\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.056767 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.056726 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc6vs\" (UniqueName: \"kubernetes.io/projected/62450e22-1ab9-4663-8960-cceab528bd29-kube-api-access-fc6vs\") pod \"insights-runtime-extractor-6gx66\" (UID: \"62450e22-1ab9-4663-8960-cceab528bd29\") " pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.056994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.056978 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vp5h\" (UniqueName: \"kubernetes.io/projected/f2d834ce-9da4-4dfe-8433-89df929dd609-kube-api-access-8vp5h\") pod \"image-registry-57899686b7-qh8vj\" (UID: \"f2d834ce-9da4-4dfe-8433-89df929dd609\") " pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.114620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.114574 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:40.130461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.130442 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" Nov 03 19:31:40.137393 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.137371 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6gx66" Nov 03 19:31:40.165332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.165004 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6c8z" Nov 03 19:31:40.220882 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.220856 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:40.395128 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.395062 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlqmb"] Nov 03 19:31:40.398763 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:40.398736 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode287d664_857c_4fb1_beb0_f248ef548609.slice/crio-92bfb1e491f657790c5e53ad633f6802915468d364fd77adffefdc6d0768027c WatchSource:0}: Error finding container 92bfb1e491f657790c5e53ad633f6802915468d364fd77adffefdc6d0768027c: Status 404 returned error can't find the container with id 92bfb1e491f657790c5e53ad633f6802915468d364fd77adffefdc6d0768027c Nov 03 19:31:40.414164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.413651 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt"] Nov 03 19:31:40.417011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.416990 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6gx66"] Nov 03 19:31:40.417827 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:40.417796 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab98eaf_1317_4f84_aab7_b3a89831ea07.slice/crio-b187e8af4600f73258dbe9fa115914d2ba95ea5325904b8ac17537d25506eece WatchSource:0}: Error finding container b187e8af4600f73258dbe9fa115914d2ba95ea5325904b8ac17537d25506eece: Status 404 returned error can't find the container with id b187e8af4600f73258dbe9fa115914d2ba95ea5325904b8ac17537d25506eece Nov 03 19:31:40.420226 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:40.420190 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62450e22_1ab9_4663_8960_cceab528bd29.slice/crio-cc7f32bb57bc3723df044332f5d7a21190956f9c026a2325933066a5cc4e2e8c WatchSource:0}: Error finding container cc7f32bb57bc3723df044332f5d7a21190956f9c026a2325933066a5cc4e2e8c: Status 404 returned error can't find the container with id cc7f32bb57bc3723df044332f5d7a21190956f9c026a2325933066a5cc4e2e8c Nov 03 19:31:40.454436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.454407 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6c8z"] Nov 03 19:31:40.466640 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:40.466610 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd524a3_fa08_43aa_8db1_71bb34dd961a.slice/crio-6371cc9ff6a38dd16cc94fed4accd657b829d07803428409a74a6862bd5e2290 WatchSource:0}: Error finding container 6371cc9ff6a38dd16cc94fed4accd657b829d07803428409a74a6862bd5e2290: Status 404 returned error can't find the container with id 6371cc9ff6a38dd16cc94fed4accd657b829d07803428409a74a6862bd5e2290 Nov 03 19:31:40.474194 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.474166 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6c8z" event={"ID":"9fd524a3-fa08-43aa-8db1-71bb34dd961a","Type":"ContainerStarted","Data":"6371cc9ff6a38dd16cc94fed4accd657b829d07803428409a74a6862bd5e2290"} Nov 03 19:31:40.475297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.475276 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" event={"ID":"4ab98eaf-1317-4f84-aab7-b3a89831ea07","Type":"ContainerStarted","Data":"b187e8af4600f73258dbe9fa115914d2ba95ea5325904b8ac17537d25506eece"} Nov 03 19:31:40.476418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.476389 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlqmb" event={"ID":"e287d664-857c-4fb1-beb0-f248ef548609","Type":"ContainerStarted","Data":"92bfb1e491f657790c5e53ad633f6802915468d364fd77adffefdc6d0768027c"} Nov 03 19:31:40.477920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.477864 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6gx66" event={"ID":"62450e22-1ab9-4663-8960-cceab528bd29","Type":"ContainerStarted","Data":"cc7f32bb57bc3723df044332f5d7a21190956f9c026a2325933066a5cc4e2e8c"} Nov 03 19:31:40.499421 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.499401 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-28czb"] Nov 03 19:31:40.501225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.501205 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:40.502529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.502506 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57899686b7-qh8vj"] Nov 03 19:31:40.505778 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:40.505756 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d834ce_9da4_4dfe_8433_89df929dd609.slice/crio-bbab30118dc0d0971ed4e5339e960cd06c3875a19e7d71f382e32ffbb555ada6 WatchSource:0}: Error finding container bbab30118dc0d0971ed4e5339e960cd06c3875a19e7d71f382e32ffbb555ada6: Status 404 returned error can't find the container with id bbab30118dc0d0971ed4e5339e960cd06c3875a19e7d71f382e32ffbb555ada6 Nov 03 19:31:40.511232 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.511211 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 19:31:40.512207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.511728 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-qx9jd\"" Nov 03 19:31:40.512207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.511770 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 19:31:40.512207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.512062 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-28czb"] Nov 03 19:31:40.550137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.550116 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxdk\" (UniqueName: \"kubernetes.io/projected/b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab-kube-api-access-lvxdk\") pod \"downloads-6dd789c8b7-28czb\" (UID: \"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab\") " pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:40.550224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.550145 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab-tmp\") pod \"downloads-6dd789c8b7-28czb\" (UID: \"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab\") " pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:40.651032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.650956 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxdk\" (UniqueName: \"kubernetes.io/projected/b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab-kube-api-access-lvxdk\") pod \"downloads-6dd789c8b7-28czb\" (UID: \"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab\") " pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:40.651032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.651009 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab-tmp\") pod \"downloads-6dd789c8b7-28czb\" (UID: \"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab\") " pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:40.651341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.651321 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab-tmp\") pod \"downloads-6dd789c8b7-28czb\" (UID: \"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab\") " pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:40.659226 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.659202 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxdk\" (UniqueName: \"kubernetes.io/projected/b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab-kube-api-access-lvxdk\") pod \"downloads-6dd789c8b7-28czb\" (UID: \"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab\") " pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:40.815433 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:40.815395 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:31:41.113943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:41.113782 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-28czb"] Nov 03 19:31:41.118965 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:41.118682 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cf18aa_f70f_4b75_ab34_c85cadb4c7ab.slice/crio-d604598d97789671df8b082a2c7f73b5d69a03850d0c0507ad949c7dd9e92722 WatchSource:0}: Error finding container d604598d97789671df8b082a2c7f73b5d69a03850d0c0507ad949c7dd9e92722: Status 404 returned error can't find the container with id d604598d97789671df8b082a2c7f73b5d69a03850d0c0507ad949c7dd9e92722 Nov 03 19:31:41.486736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:41.486651 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57899686b7-qh8vj" event={"ID":"f2d834ce-9da4-4dfe-8433-89df929dd609","Type":"ContainerStarted","Data":"51c2a8abf72fba5a417b2cfbb4cf034a81a043c5e5cad805141cf3414781c2a4"} Nov 03 19:31:41.486736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:41.486714 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57899686b7-qh8vj" event={"ID":"f2d834ce-9da4-4dfe-8433-89df929dd609","Type":"ContainerStarted","Data":"bbab30118dc0d0971ed4e5339e960cd06c3875a19e7d71f382e32ffbb555ada6"} Nov 03 19:31:41.491775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:41.491743 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6gx66" event={"ID":"62450e22-1ab9-4663-8960-cceab528bd29","Type":"ContainerStarted","Data":"0af6c7218b22007ca1bc0ee4cdb3b9d71d7436f9c227d0e2aa4a1c93bb550582"} Nov 03 19:31:41.491923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:41.491806 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6gx66" event={"ID":"62450e22-1ab9-4663-8960-cceab528bd29","Type":"ContainerStarted","Data":"d9cb63ce6b751c59d96248860e41432e614736baa12bee4c3772df41779ad61a"} Nov 03 19:31:41.495393 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:41.495364 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-28czb" event={"ID":"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab","Type":"ContainerStarted","Data":"d604598d97789671df8b082a2c7f73b5d69a03850d0c0507ad949c7dd9e92722"} Nov 03 19:31:41.504678 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:41.503949 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57899686b7-qh8vj" podStartSLOduration=2.503936498 podStartE2EDuration="2.503936498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:41.503618997 +0000 UTC m=+9.208458338" watchObservedRunningTime="2026-11-03 19:31:41.503936498 +0000 UTC m=+9.208775837" Nov 03 19:31:42.508378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:42.508353 4314 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:31:42.509429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:42.509402 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" event={"ID":"4ab98eaf-1317-4f84-aab7-b3a89831ea07","Type":"ContainerStarted","Data":"1ec8bc25aa128b2f9a5ebe8958a6c641be697769118733186d43f5fdd474fb0c"} Nov 03 19:31:42.509767 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:42.509747 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" Nov 03 19:31:42.515838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:42.515816 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" Nov 03 19:31:42.529076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:42.529027 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" podStartSLOduration=2.300857034 podStartE2EDuration="3.52901192s" totalImagesPullingTime="1.228154886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:42.527643016 +0000 UTC m=+10.232482358" watchObservedRunningTime="2026-11-03 19:31:42.52901192 +0000 UTC m=+10.233851261" Nov 03 19:31:43.485898 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.485817 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:43.486088 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.486070 4314 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:31:43.515975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.515942 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlqmb" event={"ID":"e287d664-857c-4fb1-beb0-f248ef548609","Type":"ContainerStarted","Data":"bb4e0a55a74dfdb013c7487a74a725accf0fbe382f85f5d047999bb86d0d6368"} Nov 03 19:31:43.516417 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.516003 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlqmb" event={"ID":"e287d664-857c-4fb1-beb0-f248ef548609","Type":"ContainerStarted","Data":"b9c57ec5d2e6089317a3f41079bc71639b1dc729fe1ea6d76ee743b8e0d39b48"} Nov 03 19:31:43.518640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.518610 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:43.521567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.521536 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6gx66" event={"ID":"62450e22-1ab9-4663-8960-cceab528bd29","Type":"ContainerStarted","Data":"45a52c01fa7c24e224b099b8c22361eddd9ea15b009640d7067c2a43421ba392"} Nov 03 19:31:43.539576 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.539525 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rlqmb" podStartSLOduration=2.182454606 podStartE2EDuration="4.539508648s" totalImagesPullingTime="2.357054042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:43.538898623 +0000 UTC m=+11.243737965" watchObservedRunningTime="2026-11-03 19:31:43.539508648 +0000 UTC m=+11.244347990" Nov 03 19:31:43.750340 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.750308 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlqmb_e287d664-857c-4fb1-beb0-f248ef548609/dns/0.log" Nov 03 19:31:43.930488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:43.930460 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlqmb_e287d664-857c-4fb1-beb0-f248ef548609/kube-rbac-proxy/0.log" Nov 03 19:31:44.526655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:44.526154 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6c8z" event={"ID":"9fd524a3-fa08-43aa-8db1-71bb34dd961a","Type":"ContainerStarted","Data":"5253e942835c3118de20e438459dbe2d7c88645e9325a258f949357844c112c5"} Nov 03 19:31:44.541844 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:44.541779 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6gx66" podStartSLOduration=3.000604508 podStartE2EDuration="5.54176371s" totalImagesPullingTime="2.541159202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:31:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:43.557471071 +0000 UTC m=+11.262310412" watchObservedRunningTime="2026-11-03 19:31:44.54176371 +0000 UTC m=+12.246603056" Nov 03 19:31:44.541999 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:44.541934 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c6c8z" podStartSLOduration=2.203851555 podStartE2EDuration="5.54192287s" totalImagesPullingTime="3.338071315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:44.541279605 +0000 UTC m=+12.246118946" watchObservedRunningTime="2026-11-03 19:31:44.54192287 +0000 UTC m=+12.246762210" Nov 03 19:31:44.930024 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:44.929954 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ns5zn_0225e321-66c4-4269-81d4-051793673f4d/dns-node-resolver/0.log" Nov 03 19:31:45.168220 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.168179 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:45.237489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.237415 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c94b67d4b-7wcq2"] Nov 03 19:31:45.239554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.239534 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.244626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.244593 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 19:31:45.244626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.244603 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 19:31:45.244794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.244658 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 19:31:45.244794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.244680 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 19:31:45.244903 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.244863 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-46fdn\"" Nov 03 19:31:45.244952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.244920 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 19:31:45.251383 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.251358 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c94b67d4b-7wcq2"] Nov 03 19:31:45.294614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.294572 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-service-ca\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.294749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.294635 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df31bd29-e95c-4d70-8abb-1a33b9712d2e-tmp\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.294749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.294699 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-serving-cert\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.294749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.294725 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rvg\" (UniqueName: \"kubernetes.io/projected/df31bd29-e95c-4d70-8abb-1a33b9712d2e-kube-api-access-h9rvg\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.294905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.294752 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-oauth-config\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.294905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.294766 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-oauth-serving-cert\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.294905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.294782 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-config\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.396477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.396437 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-service-ca\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.396648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.396491 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df31bd29-e95c-4d70-8abb-1a33b9712d2e-tmp\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.396648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.396531 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-serving-cert\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.396648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.396556 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rvg\" (UniqueName: \"kubernetes.io/projected/df31bd29-e95c-4d70-8abb-1a33b9712d2e-kube-api-access-h9rvg\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.396648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.396595 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-oauth-config\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.396648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.396621 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-oauth-serving-cert\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.396648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.396648 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-config\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.397176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.397124 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-service-ca\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.397293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.397174 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df31bd29-e95c-4d70-8abb-1a33b9712d2e-tmp\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.397293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.397273 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-config\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.397550 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.397531 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-oauth-serving-cert\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.399449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.399430 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-oauth-config\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.399836 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.399816 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-serving-cert\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.404498 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.404474 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rvg\" (UniqueName: \"kubernetes.io/projected/df31bd29-e95c-4d70-8abb-1a33b9712d2e-kube-api-access-h9rvg\") pod \"console-5c94b67d4b-7wcq2\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.530228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.530202 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57899686b7-qh8vj_f2d834ce-9da4-4dfe-8433-89df929dd609/registry/0.log" Nov 03 19:31:45.553483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.553458 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:45.836508 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:45.836424 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c94b67d4b-7wcq2"] Nov 03 19:31:45.839846 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:45.839813 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf31bd29_e95c_4d70_8abb_1a33b9712d2e.slice/crio-cfb857c6bae2af1b66708731f4c5bb8147555f5a8bce188b1488464a7e5dbdaf WatchSource:0}: Error finding container cfb857c6bae2af1b66708731f4c5bb8147555f5a8bce188b1488464a7e5dbdaf: Status 404 returned error can't find the container with id cfb857c6bae2af1b66708731f4c5bb8147555f5a8bce188b1488464a7e5dbdaf Nov 03 19:31:46.530847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:46.530813 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zj9jh_3fbf00d3-e30f-4399-b770-6545f69f15ce/node-ca/0.log" Nov 03 19:31:46.535300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:46.535272 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c94b67d4b-7wcq2" event={"ID":"df31bd29-e95c-4d70-8abb-1a33b9712d2e","Type":"ContainerStarted","Data":"cfb857c6bae2af1b66708731f4c5bb8147555f5a8bce188b1488464a7e5dbdaf"} Nov 03 19:31:47.130666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.130540 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c6c8z_9fd524a3-fa08-43aa-8db1-71bb34dd961a/serve-healthcheck-canary/0.log" Nov 03 19:31:47.701389 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.701355 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6"] Nov 03 19:31:47.703565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.703529 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.716315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.716066 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 19:31:47.716315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.716221 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:31:47.716561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.716455 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:31:47.717736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.717711 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:31:47.717875 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.717845 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:31:47.718730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.718709 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bp74b\"" Nov 03 19:31:47.718880 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.718858 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6"] Nov 03 19:31:47.720874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.720853 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pwnmp"] Nov 03 19:31:47.723730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.722747 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f"] Nov 03 19:31:47.723938 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.723918 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.724913 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.724892 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.734316 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.734297 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f"] Nov 03 19:31:47.751696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.751551 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:31:47.752041 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.752020 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zffgv\"" Nov 03 19:31:47.752136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.752110 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:31:47.752136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.752111 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:31:47.752250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.752117 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gs4w2\"" Nov 03 19:31:47.752250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.752242 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 19:31:47.814598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814557 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5c974e75-4f65-4ac9-913f-74b063a48dc1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.814727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814614 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-tls\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.814727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814643 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.814727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814676 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.814727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814700 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea79c75d-c7c8-452f-b84e-ec8d94449400-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814730 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814755 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62t22\" (UniqueName: \"kubernetes.io/projected/ea79c75d-c7c8-452f-b84e-ec8d94449400-kube-api-access-62t22\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814773 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814787 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hfz\" (UniqueName: \"kubernetes.io/projected/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-api-access-r9hfz\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814810 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-wtmp\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814823 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea79c75d-c7c8-452f-b84e-ec8d94449400-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814855 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-metrics-client-ca\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814880 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c974e75-4f65-4ac9-913f-74b063a48dc1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.814905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814904 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-root\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.815209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814919 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-textfile\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.815209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814951 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bffl\" (UniqueName: \"kubernetes.io/projected/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-kube-api-access-5bffl\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.815209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814978 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea79c75d-c7c8-452f-b84e-ec8d94449400-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.815209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.814999 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-sys\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.815209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.815018 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.816409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.816368 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:31:47.816515 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.816475 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 19:31:47.916256 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916215 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-root\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.916411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916264 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-textfile\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.916411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916298 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bffl\" (UniqueName: \"kubernetes.io/projected/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-kube-api-access-5bffl\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.916411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916327 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea79c75d-c7c8-452f-b84e-ec8d94449400-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.916411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916357 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-sys\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.916411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916398 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916428 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5c974e75-4f65-4ac9-913f-74b063a48dc1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916453 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-tls\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916480 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916517 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916542 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea79c75d-c7c8-452f-b84e-ec8d94449400-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916598 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916628 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62t22\" (UniqueName: \"kubernetes.io/projected/ea79c75d-c7c8-452f-b84e-ec8d94449400-kube-api-access-62t22\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916663 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.916714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916687 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hfz\" (UniqueName: \"kubernetes.io/projected/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-api-access-r9hfz\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.917138 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916725 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-wtmp\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.917138 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916750 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea79c75d-c7c8-452f-b84e-ec8d94449400-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.917138 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916786 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-metrics-client-ca\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.917138 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.916809 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c974e75-4f65-4ac9-913f-74b063a48dc1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.917598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.917555 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c974e75-4f65-4ac9-913f-74b063a48dc1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.917684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.917651 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-root\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.917921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.917903 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-textfile\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.921146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.919424 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5c974e75-4f65-4ac9-913f-74b063a48dc1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.921146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.919498 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-sys\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.921146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.919789 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea79c75d-c7c8-452f-b84e-ec8d94449400-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.921146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.919931 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-wtmp\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.921146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.920113 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.921146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.920607 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.921146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.921070 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-metrics-client-ca\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.921842 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.921792 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.922204 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.922183 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.924033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.923787 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea79c75d-c7c8-452f-b84e-ec8d94449400-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.924033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.923986 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea79c75d-c7c8-452f-b84e-ec8d94449400-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:47.925152 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.925129 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.925663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.925639 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-node-exporter-tls\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.926771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.926748 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bffl\" (UniqueName: \"kubernetes.io/projected/8cd36336-cf9e-46f7-9e42-f299f8ef0e26-kube-api-access-5bffl\") pod \"node-exporter-pwnmp\" (UID: \"8cd36336-cf9e-46f7-9e42-f299f8ef0e26\") " pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:47.926961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.926937 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hfz\" (UniqueName: \"kubernetes.io/projected/5c974e75-4f65-4ac9-913f-74b063a48dc1-kube-api-access-r9hfz\") pod \"kube-state-metrics-5dc8d7d89b-w2l2f\" (UID: \"5c974e75-4f65-4ac9-913f-74b063a48dc1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:47.927288 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:47.927268 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62t22\" (UniqueName: \"kubernetes.io/projected/ea79c75d-c7c8-452f-b84e-ec8d94449400-kube-api-access-62t22\") pod \"openshift-state-metrics-7bd7489957-wl2v6\" (UID: \"ea79c75d-c7c8-452f-b84e-ec8d94449400\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:48.019755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.019719 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" Nov 03 19:31:48.042524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.042495 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwnmp" Nov 03 19:31:48.050002 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.049980 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" Nov 03 19:31:48.053483 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:48.053449 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd36336_cf9e_46f7_9e42_f299f8ef0e26.slice/crio-4f6c5e9861d6e58629ebe5e5046c0237124fbc4eede71da5e05fcf785e63511c WatchSource:0}: Error finding container 4f6c5e9861d6e58629ebe5e5046c0237124fbc4eede71da5e05fcf785e63511c: Status 404 returned error can't find the container with id 4f6c5e9861d6e58629ebe5e5046c0237124fbc4eede71da5e05fcf785e63511c Nov 03 19:31:48.320067 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.319982 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6"] Nov 03 19:31:48.324092 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:48.324055 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea79c75d_c7c8_452f_b84e_ec8d94449400.slice/crio-47ee22d6219bc925352bbe5577e1c7a46daa0fe7f2bbbb9892a022b93b328109 WatchSource:0}: Error finding container 47ee22d6219bc925352bbe5577e1c7a46daa0fe7f2bbbb9892a022b93b328109: Status 404 returned error can't find the container with id 47ee22d6219bc925352bbe5577e1c7a46daa0fe7f2bbbb9892a022b93b328109 Nov 03 19:31:48.355643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.354777 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f"] Nov 03 19:31:48.548907 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.548868 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" event={"ID":"5c974e75-4f65-4ac9-913f-74b063a48dc1","Type":"ContainerStarted","Data":"800c17713203ea2c669e65ab75d31f0022fe0aeb1dc250189acff4ecc63fcc59"} Nov 03 19:31:48.550418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.550386 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwnmp" event={"ID":"8cd36336-cf9e-46f7-9e42-f299f8ef0e26","Type":"ContainerStarted","Data":"4f6c5e9861d6e58629ebe5e5046c0237124fbc4eede71da5e05fcf785e63511c"} Nov 03 19:31:48.553080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.552976 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" event={"ID":"ea79c75d-c7c8-452f-b84e-ec8d94449400","Type":"ContainerStarted","Data":"88cdecc87031489c810c85142a7ccefcd18873d4afa611c1390efc0d847fc592"} Nov 03 19:31:48.553080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.553026 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" event={"ID":"ea79c75d-c7c8-452f-b84e-ec8d94449400","Type":"ContainerStarted","Data":"14d8f4ba9842f739619d8cd1eb6feec9f3123af8d137c346677a34e811398393"} Nov 03 19:31:48.553080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.553047 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" event={"ID":"ea79c75d-c7c8-452f-b84e-ec8d94449400","Type":"ContainerStarted","Data":"47ee22d6219bc925352bbe5577e1c7a46daa0fe7f2bbbb9892a022b93b328109"} Nov 03 19:31:48.785122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.785084 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:31:48.790804 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.790783 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.795674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.795650 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5hwgk\"" Nov 03 19:31:48.795674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.795672 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 19:31:48.795872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.795733 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 19:31:48.795872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.795796 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 19:31:48.795872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.795871 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 19:31:48.796022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.795932 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 19:31:48.796022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.795985 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 19:31:48.796119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.796081 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 19:31:48.796178 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.796139 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 19:31:48.796225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.796196 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 19:31:48.805626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.805059 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:31:48.925390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925355 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-web-config\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925404 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925432 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925465 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925490 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-out\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925517 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txnbd\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-kube-api-access-txnbd\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925550 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925624 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925658 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925689 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925721 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925746 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.925879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.925791 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:48.938411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:48.938384 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xss4s" Nov 03 19:31:49.026321 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026288 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026337 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026367 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026426 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026475 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-web-config\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026503 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026528 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026574 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026620 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-out\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026648 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txnbd\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-kube-api-access-txnbd\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026684 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026756 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.026919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.026788 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.028282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.028244 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.030908 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.030851 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.031944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.031861 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.035626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.035527 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.036230 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.036184 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-volume\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.037709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.036950 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-out\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.037709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.037358 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.038752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.038724 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.039176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.039154 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.039275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.039201 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.040180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.040107 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-web-config\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.041537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.041513 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txnbd\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-kube-api-access-txnbd\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.046311 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.046290 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.115435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.115195 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:31:49.432717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.432692 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:31:49.438108 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:49.438077 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1a20ce_2ceb_4129_baae_c4cc56002be0.slice/crio-0576b9cf07e2ca459f1f87f81baf48f501705b804cb66767b1b8f3d18e1da993 WatchSource:0}: Error finding container 0576b9cf07e2ca459f1f87f81baf48f501705b804cb66767b1b8f3d18e1da993: Status 404 returned error can't find the container with id 0576b9cf07e2ca459f1f87f81baf48f501705b804cb66767b1b8f3d18e1da993 Nov 03 19:31:49.559853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.559772 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerStarted","Data":"0576b9cf07e2ca459f1f87f81baf48f501705b804cb66767b1b8f3d18e1da993"} Nov 03 19:31:49.561662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.561632 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c94b67d4b-7wcq2" event={"ID":"df31bd29-e95c-4d70-8abb-1a33b9712d2e","Type":"ContainerStarted","Data":"5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5"} Nov 03 19:31:49.578425 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:49.578383 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c94b67d4b-7wcq2" podStartSLOduration=1.032661883 podStartE2EDuration="4.578372006s" totalImagesPullingTime="3.545710123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:49.577832142 +0000 UTC m=+17.282671463" watchObservedRunningTime="2026-11-03 19:31:49.578372006 +0000 UTC m=+17.283211339" Nov 03 19:31:50.567456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.567418 4314 generic.go:356] "Generic (PLEG): container finished" podID="8cd36336-cf9e-46f7-9e42-f299f8ef0e26" containerID="8dd6c79dc99a1cf6a2c09407f35ba1657a9a25bfe0b54214ac129e1cbfad56f4" exitCode=0 Nov 03 19:31:50.567941 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.567507 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwnmp" event={"ID":"8cd36336-cf9e-46f7-9e42-f299f8ef0e26","Type":"ContainerDied","Data":"8dd6c79dc99a1cf6a2c09407f35ba1657a9a25bfe0b54214ac129e1cbfad56f4"} Nov 03 19:31:50.572513 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.572408 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" event={"ID":"ea79c75d-c7c8-452f-b84e-ec8d94449400","Type":"ContainerStarted","Data":"a99ceed8bbd4f97b6f52e004a8672587416514d6960703b59769dd6035522f4c"} Nov 03 19:31:50.576275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.576253 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" event={"ID":"5c974e75-4f65-4ac9-913f-74b063a48dc1","Type":"ContainerStarted","Data":"e4bbc382ef35388ab1d8aba64baeee6042fcb12caf152a994308c4552ad1644d"} Nov 03 19:31:50.576441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.576301 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" event={"ID":"5c974e75-4f65-4ac9-913f-74b063a48dc1","Type":"ContainerStarted","Data":"f916b475506fd3ae9843e32b2ab2da5b0f4fc2e13aaa64f8ad9e3cc3fa0760fb"} Nov 03 19:31:50.576441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.576332 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" event={"ID":"5c974e75-4f65-4ac9-913f-74b063a48dc1","Type":"ContainerStarted","Data":"ba3bfec9bea8dea30ed76195b654cc0c5dac37bf06fc1b069ef70e0b81cb2f66"} Nov 03 19:31:50.606343 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.606302 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wl2v6" podStartSLOduration=2.106220933 podStartE2EDuration="3.606289263s" totalImagesPullingTime="1.50006833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:50.605341178 +0000 UTC m=+18.310180536" watchObservedRunningTime="2026-11-03 19:31:50.606289263 +0000 UTC m=+18.311128603" Nov 03 19:31:50.627072 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:50.626856 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-w2l2f" podStartSLOduration=1.8228689070000001 podStartE2EDuration="3.626837302s" totalImagesPullingTime="1.803968395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:50.626056192 +0000 UTC m=+18.330895534" watchObservedRunningTime="2026-11-03 19:31:50.626837302 +0000 UTC m=+18.331676643" Nov 03 19:31:51.582137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:51.582097 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwnmp" event={"ID":"8cd36336-cf9e-46f7-9e42-f299f8ef0e26","Type":"ContainerStarted","Data":"8272018758d3ee5adddb198cd94be6e0dfa86eed05928ed27e78690ae61f5864"} Nov 03 19:31:51.582700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:51.582166 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwnmp" event={"ID":"8cd36336-cf9e-46f7-9e42-f299f8ef0e26","Type":"ContainerStarted","Data":"b12c4c66daa2fd775f2411f93a24b1cff1070dd685801f4a4bb0565507dab101"} Nov 03 19:31:51.605051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:51.605002 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pwnmp" podStartSLOduration=3.038807368 podStartE2EDuration="4.604989193s" totalImagesPullingTime="1.490837679s" totalInitContainerRuntime="75.344146ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:51.602989102 +0000 UTC m=+19.307828443" watchObservedRunningTime="2026-11-03 19:31:51.604989193 +0000 UTC m=+19.309828533" Nov 03 19:31:51.993526 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:31:51.993495 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:31:52.516227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.516194 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fv589"] Nov 03 19:31:52.519406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.519386 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:52.531375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.531352 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 19:31:52.532401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.532382 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9qw5p\"" Nov 03 19:31:52.534993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.534972 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fv589"] Nov 03 19:31:52.588001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.587969 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerID="80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74" exitCode=0 Nov 03 19:31:52.588427 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.588054 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74"} Nov 03 19:31:52.663024 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.662986 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b6e1648-d86b-4931-942a-0134faa484ae-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fv589\" (UID: \"3b6e1648-d86b-4931-942a-0134faa484ae\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:52.663204 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.663035 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhmj\" (UniqueName: \"kubernetes.io/projected/3b6e1648-d86b-4931-942a-0134faa484ae-kube-api-access-qzhmj\") pod \"monitoring-plugin-77f4f76594-fv589\" (UID: \"3b6e1648-d86b-4931-942a-0134faa484ae\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:52.763740 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.763693 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b6e1648-d86b-4931-942a-0134faa484ae-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fv589\" (UID: \"3b6e1648-d86b-4931-942a-0134faa484ae\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:52.763740 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.763745 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhmj\" (UniqueName: \"kubernetes.io/projected/3b6e1648-d86b-4931-942a-0134faa484ae-kube-api-access-qzhmj\") pod \"monitoring-plugin-77f4f76594-fv589\" (UID: \"3b6e1648-d86b-4931-942a-0134faa484ae\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:52.766705 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.766621 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b6e1648-d86b-4931-942a-0134faa484ae-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fv589\" (UID: \"3b6e1648-d86b-4931-942a-0134faa484ae\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:52.771714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.771686 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhmj\" (UniqueName: \"kubernetes.io/projected/3b6e1648-d86b-4931-942a-0134faa484ae-kube-api-access-qzhmj\") pod \"monitoring-plugin-77f4f76594-fv589\" (UID: \"3b6e1648-d86b-4931-942a-0134faa484ae\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:52.831612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:52.831570 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:53.106669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.106569 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fv589"] Nov 03 19:31:53.110751 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:53.110721 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6e1648_d86b_4931_942a_0134faa484ae.slice/crio-7d37193692dfeb5bdb15862902a03a8f490b9fbba363c9ec64fdde694087beec WatchSource:0}: Error finding container 7d37193692dfeb5bdb15862902a03a8f490b9fbba363c9ec64fdde694087beec: Status 404 returned error can't find the container with id 7d37193692dfeb5bdb15862902a03a8f490b9fbba363c9ec64fdde694087beec Nov 03 19:31:53.593688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.593642 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" event={"ID":"3b6e1648-d86b-4931-942a-0134faa484ae","Type":"ContainerStarted","Data":"7d37193692dfeb5bdb15862902a03a8f490b9fbba363c9ec64fdde694087beec"} Nov 03 19:31:53.941117 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.941043 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:31:53.944818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.944796 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:53.950161 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.949875 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.951472 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.951497 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.951762 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.951856 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ahgouu4ist57l\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.952043 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.952152 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w2tdz\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.952226 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.952358 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.952417 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:31:53.952654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.952641 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:31:53.953113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.952987 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:31:53.953113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.953023 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:31:53.954561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.953734 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:31:53.954561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.953971 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 19:31:53.955200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.955167 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:31:53.955609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.955569 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:31:53.959168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:53.959056 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:31:54.080679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.080633 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.080841 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.080704 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.080887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.080829 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.080927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.080887 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081023 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.080998 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081106 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081041 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081106 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081068 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081208 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081127 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081208 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081164 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hch9q\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-kube-api-access-hch9q\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081240 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081266 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081402 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081322 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081402 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081351 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081402 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081379 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081403 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081426 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081456 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081482 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081508 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.081554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.081545 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.182895 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.182858 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.182912 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.182941 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.182978 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183002 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183030 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183057 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183096 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183123 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183176 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183206 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183254 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183287 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183316 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183341 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183366 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183389 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hch9q\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-kube-api-access-hch9q\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183431 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183453 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.183948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183516 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.184166 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.184089 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.185695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.183311 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.185695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.185395 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.185695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.185497 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.186018 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.185994 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.186461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.186422 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.189624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.186683 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.189624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.187164 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.189624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.187533 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.189624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.187807 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.189624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.189557 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.193642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.193189 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.193642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.193296 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.193642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.193544 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.193642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.193598 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.193861 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.193760 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.194186 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.194166 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.194390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.194356 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.194467 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.194444 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hch9q\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-kube-api-access-hch9q\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.194641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.194576 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.283004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.282736 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:31:54.532349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.532317 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rlqmb" Nov 03 19:31:54.582931 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.582898 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:31:54.586836 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:31:54.586803 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb65768d_6e8c_4868_bd82_f3f5b149eeb2.slice/crio-c18e3e1a55f7da586b212366807642f4c130f6c89bf47e17d2d1773efde68719 WatchSource:0}: Error finding container c18e3e1a55f7da586b212366807642f4c130f6c89bf47e17d2d1773efde68719: Status 404 returned error can't find the container with id c18e3e1a55f7da586b212366807642f4c130f6c89bf47e17d2d1773efde68719 Nov 03 19:31:54.601760 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:54.601652 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerStarted","Data":"c18e3e1a55f7da586b212366807642f4c130f6c89bf47e17d2d1773efde68719"} Nov 03 19:31:55.173781 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.173749 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57899686b7-qh8vj" Nov 03 19:31:55.554117 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.554076 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:55.554284 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.554265 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:55.560075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.560052 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:55.610493 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.610460 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" event={"ID":"3b6e1648-d86b-4931-942a-0134faa484ae","Type":"ContainerStarted","Data":"5bc4ba8c4f373f97e6ed2286d3cb3f416594f5ee72084d6e1365f1011b7ecf36"} Nov 03 19:31:55.610956 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.610683 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:55.612885 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.612851 4314 generic.go:356] "Generic (PLEG): container finished" podID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" exitCode=0 Nov 03 19:31:55.613002 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.612976 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34"} Nov 03 19:31:55.616386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.616295 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" Nov 03 19:31:55.618404 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.618382 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerStarted","Data":"add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1"} Nov 03 19:31:55.618565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.618441 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerStarted","Data":"74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc"} Nov 03 19:31:55.618565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.618469 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerStarted","Data":"196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168"} Nov 03 19:31:55.618565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.618498 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerStarted","Data":"d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688"} Nov 03 19:31:55.618565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.618551 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerStarted","Data":"2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c"} Nov 03 19:31:55.622722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.622679 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:31:55.643459 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:55.643406 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fv589" podStartSLOduration=1.9755989889999999 podStartE2EDuration="3.643390337s" totalImagesPullingTime="1.667791348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:55.626633875 +0000 UTC m=+23.331473213" watchObservedRunningTime="2026-11-03 19:31:55.643390337 +0000 UTC m=+23.348229673" Nov 03 19:31:56.627857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:56.627773 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerStarted","Data":"5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5"} Nov 03 19:31:56.655723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:56.655666 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.063091319 podStartE2EDuration="8.655641193s" totalImagesPullingTime="5.580608131s" totalInitContainerRuntime="11.941743ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:56.652934043 +0000 UTC m=+24.357773384" watchObservedRunningTime="2026-11-03 19:31:56.655641193 +0000 UTC m=+24.360480533" Nov 03 19:31:59.646427 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:59.646346 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerStarted","Data":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} Nov 03 19:31:59.646948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:31:59.646441 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerStarted","Data":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} Nov 03 19:32:00.653692 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:00.653649 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-28czb" event={"ID":"b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab","Type":"ContainerStarted","Data":"09ea95151d25285de7af3146fb191343a6ba0b566aca69f44911a565461adc06"} Nov 03 19:32:00.654196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:00.653984 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:32:00.666885 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:00.666864 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-28czb" Nov 03 19:32:00.673598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:00.673533 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-28czb" podStartSLOduration=1.991934272 podStartE2EDuration="20.673521785s" totalImagesPullingTime="18.681587513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:00.671325412 +0000 UTC m=+28.376164753" watchObservedRunningTime="2026-11-03 19:32:00.673521785 +0000 UTC m=+28.378361126" Nov 03 19:32:02.701452 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:02.701408 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerStarted","Data":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} Nov 03 19:32:02.701864 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:02.701516 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerStarted","Data":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} Nov 03 19:32:03.710207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:03.710170 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerStarted","Data":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} Nov 03 19:32:03.710711 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:03.710249 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerStarted","Data":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} Nov 03 19:32:03.745608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:03.743999 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.128101407 podStartE2EDuration="10.743980453s" totalImagesPullingTime="6.593039854s" totalInitContainerRuntime="22.839192ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:03.740274112 +0000 UTC m=+31.445113453" watchObservedRunningTime="2026-11-03 19:32:03.743980453 +0000 UTC m=+31.448819796" Nov 03 19:32:04.283499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:04.283462 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:06.960691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:06.960658 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pp285" Nov 03 19:32:07.858260 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:07.858220 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d799d7c5b-b55bb"] Nov 03 19:32:07.884857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:07.884821 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d799d7c5b-b55bb"] Nov 03 19:32:07.884994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:07.884964 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:07.894770 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:07.894422 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 19:32:08.025028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.024994 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-oauth-serving-cert\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.025473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.025046 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-trusted-ca-bundle\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.025473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.025130 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-oauth-config\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.025473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.025177 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935cd956-ac7a-426a-9cb3-20dc0c96582f-tmp\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.025473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.025203 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-serving-cert\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.025473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.025291 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-config\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.025473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.025390 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-service-ca\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.025473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.025429 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cjtw\" (UniqueName: \"kubernetes.io/projected/935cd956-ac7a-426a-9cb3-20dc0c96582f-kube-api-access-2cjtw\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.126985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.126915 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cjtw\" (UniqueName: \"kubernetes.io/projected/935cd956-ac7a-426a-9cb3-20dc0c96582f-kube-api-access-2cjtw\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.126985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.126964 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-oauth-serving-cert\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127029 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-trusted-ca-bundle\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127054 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-oauth-config\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127215 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127184 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935cd956-ac7a-426a-9cb3-20dc0c96582f-tmp\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127234 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-serving-cert\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127297 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-config\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127381 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127370 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-service-ca\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127625 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935cd956-ac7a-426a-9cb3-20dc0c96582f-tmp\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.127918 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.127892 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-oauth-serving-cert\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.128133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.128110 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-config\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.128326 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.128297 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-service-ca\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.128544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.128523 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-trusted-ca-bundle\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.129828 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.129804 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-oauth-config\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.129934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.129844 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-serving-cert\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.134663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.134644 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cjtw\" (UniqueName: \"kubernetes.io/projected/935cd956-ac7a-426a-9cb3-20dc0c96582f-kube-api-access-2cjtw\") pod \"console-5d799d7c5b-b55bb\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.212543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.212519 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:08.510691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.510662 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d799d7c5b-b55bb"] Nov 03 19:32:08.513443 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:32:08.513411 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935cd956_ac7a_426a_9cb3_20dc0c96582f.slice/crio-2905c1edb99fc0413bd6c4c811529a6180a13e44c9092cdce8d23fec73dddeef WatchSource:0}: Error finding container 2905c1edb99fc0413bd6c4c811529a6180a13e44c9092cdce8d23fec73dddeef: Status 404 returned error can't find the container with id 2905c1edb99fc0413bd6c4c811529a6180a13e44c9092cdce8d23fec73dddeef Nov 03 19:32:08.732653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.732617 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799d7c5b-b55bb" event={"ID":"935cd956-ac7a-426a-9cb3-20dc0c96582f","Type":"ContainerStarted","Data":"a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c"} Nov 03 19:32:08.732811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.732667 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799d7c5b-b55bb" event={"ID":"935cd956-ac7a-426a-9cb3-20dc0c96582f","Type":"ContainerStarted","Data":"2905c1edb99fc0413bd6c4c811529a6180a13e44c9092cdce8d23fec73dddeef"} Nov 03 19:32:08.750143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:08.750097 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d799d7c5b-b55bb" podStartSLOduration=1.750085696 podStartE2EDuration="1.750085696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:08.7488482 +0000 UTC m=+36.453687541" watchObservedRunningTime="2026-11-03 19:32:08.750085696 +0000 UTC m=+36.454925035" Nov 03 19:32:18.213410 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:18.213332 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:18.213410 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:18.213383 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:18.218137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:18.218114 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:18.774681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:18.774655 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:32:18.817048 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:18.817014 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c94b67d4b-7wcq2"] Nov 03 19:32:21.988530 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:21.988484 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:32:38.138437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:38.138398 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:38.186531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:38.186508 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:38.858687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:38.858658 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:43.837627 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:43.837555 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c94b67d4b-7wcq2" podUID="df31bd29-e95c-4d70-8abb-1a33b9712d2e" containerName="console" containerID="cri-o://5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5" gracePeriod=15 Nov 03 19:32:44.110524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.110464 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:32:44.225054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225026 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df31bd29-e95c-4d70-8abb-1a33b9712d2e-kube-api-access-h9rvg\" (UniqueName: \"kubernetes.io/projected/df31bd29-e95c-4d70-8abb-1a33b9712d2e-kube-api-access-h9rvg\") pod \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " Nov 03 19:32:44.225217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225078 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df31bd29-e95c-4d70-8abb-1a33b9712d2e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/df31bd29-e95c-4d70-8abb-1a33b9712d2e-tmp\") pod \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " Nov 03 19:32:44.225217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225097 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-service-ca\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-service-ca\") pod \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " Nov 03 19:32:44.225217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225120 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-config\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-config\") pod \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " Nov 03 19:32:44.225217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225147 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-oauth-serving-cert\") pod \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " Nov 03 19:32:44.225217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225184 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-serving-cert\") pod \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " Nov 03 19:32:44.225484 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225254 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-oauth-config\") pod \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\" (UID: \"df31bd29-e95c-4d70-8abb-1a33b9712d2e\") " Nov 03 19:32:44.225484 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225385 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df31bd29-e95c-4d70-8abb-1a33b9712d2e-tmp" pod "df31bd29-e95c-4d70-8abb-1a33b9712d2e" (UID: "df31bd29-e95c-4d70-8abb-1a33b9712d2e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:44.225618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225520 4314 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df31bd29-e95c-4d70-8abb-1a33b9712d2e-tmp\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:44.225618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225531 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-service-ca" pod "df31bd29-e95c-4d70-8abb-1a33b9712d2e" (UID: "df31bd29-e95c-4d70-8abb-1a33b9712d2e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:44.225618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225578 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-oauth-serving-cert" pod "df31bd29-e95c-4d70-8abb-1a33b9712d2e" (UID: "df31bd29-e95c-4d70-8abb-1a33b9712d2e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:44.225750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.225635 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-config" pod "df31bd29-e95c-4d70-8abb-1a33b9712d2e" (UID: "df31bd29-e95c-4d70-8abb-1a33b9712d2e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:44.227384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.227361 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-serving-cert" pod "df31bd29-e95c-4d70-8abb-1a33b9712d2e" (UID: "df31bd29-e95c-4d70-8abb-1a33b9712d2e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:44.227491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.227379 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-oauth-config" pod "df31bd29-e95c-4d70-8abb-1a33b9712d2e" (UID: "df31bd29-e95c-4d70-8abb-1a33b9712d2e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:44.227491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.227397 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df31bd29-e95c-4d70-8abb-1a33b9712d2e-kube-api-access-h9rvg" pod "df31bd29-e95c-4d70-8abb-1a33b9712d2e" (UID: "df31bd29-e95c-4d70-8abb-1a33b9712d2e"). InnerVolumeSpecName "kube-api-access-h9rvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:44.326102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.326077 4314 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-oauth-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:44.326102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.326101 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rvg\" (UniqueName: \"kubernetes.io/projected/df31bd29-e95c-4d70-8abb-1a33b9712d2e-kube-api-access-h9rvg\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:44.326271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.326112 4314 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:44.326271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.326121 4314 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:44.326271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.326130 4314 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df31bd29-e95c-4d70-8abb-1a33b9712d2e-oauth-serving-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:44.326271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.326138 4314 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df31bd29-e95c-4d70-8abb-1a33b9712d2e-console-serving-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:44.866055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.866021 4314 generic.go:356] "Generic (PLEG): container finished" podID="df31bd29-e95c-4d70-8abb-1a33b9712d2e" containerID="5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5" exitCode=0 Nov 03 19:32:44.866462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.866064 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c94b67d4b-7wcq2" Nov 03 19:32:44.866462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.866093 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c94b67d4b-7wcq2" event={"ID":"df31bd29-e95c-4d70-8abb-1a33b9712d2e","Type":"ContainerDied","Data":"5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5"} Nov 03 19:32:44.866462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.866140 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c94b67d4b-7wcq2" event={"ID":"df31bd29-e95c-4d70-8abb-1a33b9712d2e","Type":"ContainerDied","Data":"cfb857c6bae2af1b66708731f4c5bb8147555f5a8bce188b1488464a7e5dbdaf"} Nov 03 19:32:44.866462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.866160 4314 scope.go:122] "RemoveContainer" containerID="5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5" Nov 03 19:32:44.874171 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.874157 4314 scope.go:122] "RemoveContainer" containerID="5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5" Nov 03 19:32:44.874436 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:44.874418 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5\": container with ID starting with 5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5 not found: ID does not exist" containerID="5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5" Nov 03 19:32:44.874500 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.874445 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5"} err="failed to get container status \"5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5\": rpc error: code = NotFound desc = could not find container \"5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5\": container with ID starting with 5617143f1de2afc62408c15d84c084bbf6273b7045ae8bc370885423b373e4e5 not found: ID does not exist" Nov 03 19:32:44.881221 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.881200 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c94b67d4b-7wcq2"] Nov 03 19:32:44.887227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:44.887207 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c94b67d4b-7wcq2"] Nov 03 19:32:46.424916 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:46.424886 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df31bd29-e95c-4d70-8abb-1a33b9712d2e" path="/var/lib/kubelet/pods/df31bd29-e95c-4d70-8abb-1a33b9712d2e/volumes" Nov 03 19:32:48.013819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.013783 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:32:48.014211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.014182 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="alertmanager" containerID="cri-o://2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c" gracePeriod=120 Nov 03 19:32:48.014437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.014260 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-web" containerID="cri-o://196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168" gracePeriod=120 Nov 03 19:32:48.014437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.014297 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="prom-label-proxy" containerID="cri-o://5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5" gracePeriod=120 Nov 03 19:32:48.014437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.014279 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy" containerID="cri-o://74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc" gracePeriod=120 Nov 03 19:32:48.014437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.014246 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-metric" containerID="cri-o://add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1" gracePeriod=120 Nov 03 19:32:48.014437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.014346 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="config-reloader" containerID="cri-o://d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688" gracePeriod=120 Nov 03 19:32:48.884010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.883976 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerID="5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5" exitCode=0 Nov 03 19:32:48.884010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884002 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerID="add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1" exitCode=0 Nov 03 19:32:48.884010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884008 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerID="74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc" exitCode=0 Nov 03 19:32:48.884010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884014 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerID="d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688" exitCode=0 Nov 03 19:32:48.884010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884019 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerID="2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c" exitCode=0 Nov 03 19:32:48.884305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884044 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5"} Nov 03 19:32:48.884305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884104 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1"} Nov 03 19:32:48.884305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884131 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc"} Nov 03 19:32:48.884305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884156 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688"} Nov 03 19:32:48.884305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:48.884182 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c"} Nov 03 19:32:49.272936 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.272908 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:32:49.366875 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.366847 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-web\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.366882 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-tls-assets\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-tls-assets\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.366921 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-main-db\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367034 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-kube-api-access-txnbd\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-kube-api-access-txnbd\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367072 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-metrics-client-ca\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367103 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-main-tls\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367131 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-out\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367208 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-trusted-ca-bundle\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367232 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-main-db" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:49.367429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367244 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367289 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-cluster-tls-config\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367339 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-web-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-web-config\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367386 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-volume\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-volume\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367420 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\" (UID: \"5e1a20ce-2ceb-4129-baae-c4cc56002be0\") " Nov 03 19:32:49.367819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367646 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-trusted-ca-bundle" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:49.367819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367681 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-metrics-client-ca" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:49.367819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.367703 4314 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-main-db\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.369763 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.369724 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-tls-assets" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:49.369926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.369901 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-web" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:49.370267 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.370237 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-kube-api-access-txnbd" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "kube-api-access-txnbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:49.370712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.370660 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-main-tls" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:49.370814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.370786 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-volume" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:49.370814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.370798 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-out" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:49.371638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.371612 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-metric" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:49.372227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.372199 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:49.374046 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.374023 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-cluster-tls-config" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:49.380297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.380277 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-web-config" pod "5e1a20ce-2ceb-4129-baae-c4cc56002be0" (UID: "5e1a20ce-2ceb-4129-baae-c4cc56002be0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:49.468412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468344 4314 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-web-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468371 4314 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468381 4314 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468392 4314 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468401 4314 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-tls-assets\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468412 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txnbd\" (UniqueName: \"kubernetes.io/projected/5e1a20ce-2ceb-4129-baae-c4cc56002be0-kube-api-access-txnbd\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468421 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-metrics-client-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468430 4314 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-main-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468438 4314 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5e1a20ce-2ceb-4129-baae-c4cc56002be0-config-out\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468448 4314 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e1a20ce-2ceb-4129-baae-c4cc56002be0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468456 4314 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.468718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.468466 4314 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5e1a20ce-2ceb-4129-baae-c4cc56002be0-cluster-tls-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:49.891330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.891296 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerID="196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168" exitCode=0 Nov 03 19:32:49.891591 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.891372 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168"} Nov 03 19:32:49.891591 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.891389 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:32:49.891591 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.891438 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5e1a20ce-2ceb-4129-baae-c4cc56002be0","Type":"ContainerDied","Data":"0576b9cf07e2ca459f1f87f81baf48f501705b804cb66767b1b8f3d18e1da993"} Nov 03 19:32:49.891591 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.891471 4314 scope.go:122] "RemoveContainer" containerID="5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5" Nov 03 19:32:49.899801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.899780 4314 scope.go:122] "RemoveContainer" containerID="add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1" Nov 03 19:32:49.906479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.906462 4314 scope.go:122] "RemoveContainer" containerID="74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc" Nov 03 19:32:49.913191 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.913170 4314 scope.go:122] "RemoveContainer" containerID="196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168" Nov 03 19:32:49.914946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.914923 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:32:49.920403 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.920382 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:32:49.921396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.921381 4314 scope.go:122] "RemoveContainer" containerID="d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688" Nov 03 19:32:49.927673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.927654 4314 scope.go:122] "RemoveContainer" containerID="2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c" Nov 03 19:32:49.934113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.934097 4314 scope.go:122] "RemoveContainer" containerID="80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74" Nov 03 19:32:49.941699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.941684 4314 scope.go:122] "RemoveContainer" containerID="5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5" Nov 03 19:32:49.941987 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:49.941951 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5\": container with ID starting with 5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5 not found: ID does not exist" containerID="5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5" Nov 03 19:32:49.942071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.941994 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5"} err="failed to get container status \"5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5\": rpc error: code = NotFound desc = could not find container \"5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5\": container with ID starting with 5e6cfcf653bd95891d83530993ee4229ce5ab62f709570d11b18f9b75d82c6a5 not found: ID does not exist" Nov 03 19:32:49.942071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.942013 4314 scope.go:122] "RemoveContainer" containerID="add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1" Nov 03 19:32:49.942561 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:49.942259 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1\": container with ID starting with add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1 not found: ID does not exist" containerID="add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1" Nov 03 19:32:49.942561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.942368 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1"} err="failed to get container status \"add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1\": rpc error: code = NotFound desc = could not find container \"add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1\": container with ID starting with add598007fcd05680e683edbb87a517da085d2b0cbdc793eb9cf5284255a15c1 not found: ID does not exist" Nov 03 19:32:49.942561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.942387 4314 scope.go:122] "RemoveContainer" containerID="74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc" Nov 03 19:32:49.943124 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:49.942770 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc\": container with ID starting with 74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc not found: ID does not exist" containerID="74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc" Nov 03 19:32:49.943124 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.942797 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc"} err="failed to get container status \"74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc\": rpc error: code = NotFound desc = could not find container \"74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc\": container with ID starting with 74bf3cc35c4e0d443ffc5ff7361d09d539738abf32f227c14d0248abf3ae16bc not found: ID does not exist" Nov 03 19:32:49.943124 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.942823 4314 scope.go:122] "RemoveContainer" containerID="196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168" Nov 03 19:32:49.943124 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:49.943107 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168\": container with ID starting with 196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168 not found: ID does not exist" containerID="196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168" Nov 03 19:32:49.943333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.943137 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168"} err="failed to get container status \"196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168\": rpc error: code = NotFound desc = could not find container \"196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168\": container with ID starting with 196b659f65139b8522b5d732e1d12f5ebd38c8777b8bde436d3397ecd9acd168 not found: ID does not exist" Nov 03 19:32:49.943333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.943159 4314 scope.go:122] "RemoveContainer" containerID="d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688" Nov 03 19:32:49.943441 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:49.943408 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688\": container with ID starting with d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688 not found: ID does not exist" containerID="d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688" Nov 03 19:32:49.943495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.943433 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688"} err="failed to get container status \"d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688\": rpc error: code = NotFound desc = could not find container \"d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688\": container with ID starting with d39580ebe5a177a2eab07959480e9317d0e340b3e7c1aab658a663f55ed95688 not found: ID does not exist" Nov 03 19:32:49.943495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.943453 4314 scope.go:122] "RemoveContainer" containerID="2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c" Nov 03 19:32:49.943750 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:49.943730 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c\": container with ID starting with 2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c not found: ID does not exist" containerID="2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c" Nov 03 19:32:49.943814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.943760 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c"} err="failed to get container status \"2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c\": rpc error: code = NotFound desc = could not find container \"2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c\": container with ID starting with 2f98c868d212fa1dde4c1d24300d205af42b570bf0c287045579fa2157cdfb1c not found: ID does not exist" Nov 03 19:32:49.943814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.943781 4314 scope.go:122] "RemoveContainer" containerID="80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74" Nov 03 19:32:49.944038 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:49.944014 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74\": container with ID starting with 80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74 not found: ID does not exist" containerID="80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74" Nov 03 19:32:49.944078 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:49.944043 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74"} err="failed to get container status \"80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74\": rpc error: code = NotFound desc = could not find container \"80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74\": container with ID starting with 80e6816d1f9ec29eeb1a79d29604c01ef57e46ae8a26fbcf4bca5670d495ea74 not found: ID does not exist" Nov 03 19:32:50.425441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:50.425402 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" path="/var/lib/kubelet/pods/5e1a20ce-2ceb-4129-baae-c4cc56002be0/volumes" Nov 03 19:32:51.988668 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:51.988636 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:32:52.237249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.237219 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:32:52.237548 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.237519 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="prometheus" containerID="cri-o://5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" gracePeriod=600 Nov 03 19:32:52.237709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.237546 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy" containerID="cri-o://4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" gracePeriod=600 Nov 03 19:32:52.237709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.237566 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" gracePeriod=600 Nov 03 19:32:52.237709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.237603 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="config-reloader" containerID="cri-o://c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" gracePeriod=600 Nov 03 19:32:52.237709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.237607 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-web" containerID="cri-o://0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" gracePeriod=600 Nov 03 19:32:52.239213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.238184 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="thanos-sidecar" containerID="cri-o://85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" gracePeriod=600 Nov 03 19:32:52.472160 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.472136 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:52.592345 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592314 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-web-config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-web-config\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592345 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592346 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-kube-rbac-proxy\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592368 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-metrics-client-certs\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592395 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-kube-api-access-hch9q\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-kube-api-access-hch9q\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592425 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-db\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592446 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-metrics-client-ca\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592497 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592525 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592596 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-0\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592641 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592675 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-grpc-tls\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592714 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-tls\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592744 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-1\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592777 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-trusted-ca-bundle\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.592845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592814 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-serving-certs-ca-bundle\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.593200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592876 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-kubelet-serving-ca-bundle\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.593200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592905 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-2\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.593200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592934 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-thanos-prometheus-http-client-file\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.593200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.592963 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-tls-assets\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-tls-assets\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.593200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.593001 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config-out\") pod \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\" (UID: \"fb65768d-6e8c-4868-bd82-f3f5b149eeb2\") " Nov 03 19:32:52.593466 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.593204 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-metrics-client-ca" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:52.594281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.593575 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-2" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:52.594281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.593806 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-serving-certs-ca-bundle" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:52.594281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.594010 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-db" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:52.594281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.594039 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-1" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:52.594609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.594352 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-kubelet-serving-ca-bundle" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:52.594609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.594518 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-trusted-ca-bundle" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:52.594938 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.594910 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-0" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:32:52.595143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.595120 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-metrics-client-certs" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.595319 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.595299 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-kube-api-access-hch9q" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "kube-api-access-hch9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:52.595481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.595460 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-kube-rbac-proxy" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.595997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.595970 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-tls" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.596448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.596411 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-thanos-sidecar-tls" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.597064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.597018 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.597148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.597072 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-tls-assets" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:32:52.597372 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.597349 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-grpc-tls" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.597558 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.597543 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.597738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.597723 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config-out" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:32:52.597792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.597740 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-thanos-prometheus-http-client-file" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.605879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.605861 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-web-config" pod "fb65768d-6e8c-4868-bd82-f3f5b149eeb2" (UID: "fb65768d-6e8c-4868-bd82-f3f5b149eeb2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:32:52.694330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694305 4314 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-grpc-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694330 4314 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694341 4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694350 4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694359 4314 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694370 4314 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694379 4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694388 4314 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694396 4314 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-tls-assets\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694404 4314 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config-out\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694414 4314 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-web-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694422 4314 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-kube-rbac-proxy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694431 4314 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-metrics-client-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694439 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hch9q\" (UniqueName: \"kubernetes.io/projected/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-kube-api-access-hch9q\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694448 4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-db\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694457 4314 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-configmap-metrics-client-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694465 4314 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694473 4314 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.694477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694481 4314 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.695023 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.694490 4314 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb65768d-6e8c-4868-bd82-f3f5b149eeb2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:32:52.905534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905464 4314 generic.go:356] "Generic (PLEG): container finished" podID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" exitCode=0 Nov 03 19:32:52.905534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905488 4314 generic.go:356] "Generic (PLEG): container finished" podID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" exitCode=0 Nov 03 19:32:52.905534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905495 4314 generic.go:356] "Generic (PLEG): container finished" podID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" exitCode=0 Nov 03 19:32:52.905534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905500 4314 generic.go:356] "Generic (PLEG): container finished" podID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" exitCode=0 Nov 03 19:32:52.905534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905507 4314 generic.go:356] "Generic (PLEG): container finished" podID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" exitCode=0 Nov 03 19:32:52.905534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905512 4314 generic.go:356] "Generic (PLEG): container finished" podID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" exitCode=0 Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905545 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905545 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905623 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905653 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905681 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905688 4314 scope.go:122] "RemoveContainer" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905708 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905739 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} Nov 03 19:32:52.905863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.905764 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb65768d-6e8c-4868-bd82-f3f5b149eeb2","Type":"ContainerDied","Data":"c18e3e1a55f7da586b212366807642f4c130f6c89bf47e17d2d1773efde68719"} Nov 03 19:32:52.913353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.913333 4314 scope.go:122] "RemoveContainer" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.920061 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.920045 4314 scope.go:122] "RemoveContainer" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.926346 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.926327 4314 scope.go:122] "RemoveContainer" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.929409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.929389 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:32:52.934047 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.934022 4314 scope.go:122] "RemoveContainer" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.934483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.934464 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:32:52.940356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.940341 4314 scope.go:122] "RemoveContainer" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.947392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.947377 4314 scope.go:122] "RemoveContainer" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.953751 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.953734 4314 scope.go:122] "RemoveContainer" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.954012 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:52.953997 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": container with ID starting with d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918 not found: ID does not exist" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.954051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954022 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} err="failed to get container status \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": rpc error: code = NotFound desc = could not find container \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": container with ID starting with d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918 not found: ID does not exist" Nov 03 19:32:52.954051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954038 4314 scope.go:122] "RemoveContainer" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.954259 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:52.954241 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": container with ID starting with 4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723 not found: ID does not exist" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.954296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954264 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} err="failed to get container status \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": rpc error: code = NotFound desc = could not find container \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": container with ID starting with 4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723 not found: ID does not exist" Nov 03 19:32:52.954296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954277 4314 scope.go:122] "RemoveContainer" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.954478 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:52.954464 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": container with ID starting with 0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088 not found: ID does not exist" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.954512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954482 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} err="failed to get container status \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": rpc error: code = NotFound desc = could not find container \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": container with ID starting with 0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088 not found: ID does not exist" Nov 03 19:32:52.954512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954494 4314 scope.go:122] "RemoveContainer" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.954946 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:52.954928 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": container with ID starting with 85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570 not found: ID does not exist" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.955031 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954948 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} err="failed to get container status \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": rpc error: code = NotFound desc = could not find container \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": container with ID starting with 85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570 not found: ID does not exist" Nov 03 19:32:52.955031 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.954962 4314 scope.go:122] "RemoveContainer" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.955185 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:52.955165 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": container with ID starting with c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec not found: ID does not exist" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.955259 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.955194 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} err="failed to get container status \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": rpc error: code = NotFound desc = could not find container \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": container with ID starting with c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec not found: ID does not exist" Nov 03 19:32:52.955259 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.955215 4314 scope.go:122] "RemoveContainer" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.955750 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:52.955721 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": container with ID starting with 5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc not found: ID does not exist" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.955858 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.955755 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} err="failed to get container status \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": rpc error: code = NotFound desc = could not find container \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": container with ID starting with 5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc not found: ID does not exist" Nov 03 19:32:52.955858 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.955776 4314 scope.go:122] "RemoveContainer" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.956075 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:32:52.956056 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": container with ID starting with 0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34 not found: ID does not exist" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.956147 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956081 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34"} err="failed to get container status \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": rpc error: code = NotFound desc = could not find container \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": container with ID starting with 0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34 not found: ID does not exist" Nov 03 19:32:52.956147 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956099 4314 scope.go:122] "RemoveContainer" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.956373 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956353 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} err="failed to get container status \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": rpc error: code = NotFound desc = could not find container \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": container with ID starting with d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918 not found: ID does not exist" Nov 03 19:32:52.956416 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956374 4314 scope.go:122] "RemoveContainer" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.956628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956605 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} err="failed to get container status \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": rpc error: code = NotFound desc = could not find container \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": container with ID starting with 4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723 not found: ID does not exist" Nov 03 19:32:52.956706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956629 4314 scope.go:122] "RemoveContainer" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.956840 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956812 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:32:52.956884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956846 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} err="failed to get container status \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": rpc error: code = NotFound desc = could not find container \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": container with ID starting with 0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088 not found: ID does not exist" Nov 03 19:32:52.956884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.956863 4314 scope.go:122] "RemoveContainer" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.957093 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957079 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy" Nov 03 19:32:52.957171 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957086 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} err="failed to get container status \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": rpc error: code = NotFound desc = could not find container \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": container with ID starting with 85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570 not found: ID does not exist" Nov 03 19:32:52.957171 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957113 4314 scope.go:122] "RemoveContainer" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.957171 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957095 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957185 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-thanos" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957197 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-thanos" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957224 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="alertmanager" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957232 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="alertmanager" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957242 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="prometheus" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957249 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="prometheus" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957260 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957267 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957283 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="init-config-reloader" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957290 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="init-config-reloader" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957300 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="thanos-sidecar" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957308 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="thanos-sidecar" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957318 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-web" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957326 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-web" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957326 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} err="failed to get container status \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": rpc error: code = NotFound desc = could not find container \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": container with ID starting with c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec not found: ID does not exist" Nov 03 19:32:52.957332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957339 4314 cpu_manager.go:414] "containerMap: removing container" podUID="df31bd29-e95c-4d70-8abb-1a33b9712d2e" containerName="console" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957346 4314 scope.go:122] "RemoveContainer" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957347 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df31bd29-e95c-4d70-8abb-1a33b9712d2e" containerName="console" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957394 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957401 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957413 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-metric" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957418 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-metric" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957425 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="init-config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957432 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="init-config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957442 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="prom-label-proxy" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957449 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="prom-label-proxy" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957465 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957472 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957482 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-web" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957488 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-web" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957562 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-web" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957575 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957604 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-thanos" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957614 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="alertmanager" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957620 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="prometheus" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957625 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957631 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="thanos-sidecar" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957636 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="kube-rbac-proxy-web" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957642 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="df31bd29-e95c-4d70-8abb-1a33b9712d2e" containerName="console" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957648 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957654 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="kube-rbac-proxy-metric" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957659 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1a20ce-2ceb-4129-baae-c4cc56002be0" containerName="prom-label-proxy" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957653 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} err="failed to get container status \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": rpc error: code = NotFound desc = could not find container \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": container with ID starting with 5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc not found: ID does not exist" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957680 4314 scope.go:122] "RemoveContainer" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957665 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" containerName="config-reloader" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957923 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34"} err="failed to get container status \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": rpc error: code = NotFound desc = could not find container \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": container with ID starting with 0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34 not found: ID does not exist" Nov 03 19:32:52.957989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.957940 4314 scope.go:122] "RemoveContainer" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958146 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} err="failed to get container status \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": rpc error: code = NotFound desc = could not find container \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": container with ID starting with d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918 not found: ID does not exist" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958163 4314 scope.go:122] "RemoveContainer" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958345 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} err="failed to get container status \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": rpc error: code = NotFound desc = could not find container \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": container with ID starting with 4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723 not found: ID does not exist" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958361 4314 scope.go:122] "RemoveContainer" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958515 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} err="failed to get container status \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": rpc error: code = NotFound desc = could not find container \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": container with ID starting with 0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088 not found: ID does not exist" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958531 4314 scope.go:122] "RemoveContainer" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958734 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} err="failed to get container status \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": rpc error: code = NotFound desc = could not find container \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": container with ID starting with 85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570 not found: ID does not exist" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958750 4314 scope.go:122] "RemoveContainer" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958955 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} err="failed to get container status \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": rpc error: code = NotFound desc = could not find container \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": container with ID starting with c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec not found: ID does not exist" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.958969 4314 scope.go:122] "RemoveContainer" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959189 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} err="failed to get container status \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": rpc error: code = NotFound desc = could not find container \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": container with ID starting with 5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc not found: ID does not exist" Nov 03 19:32:52.959368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959205 4314 scope.go:122] "RemoveContainer" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.959724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959405 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34"} err="failed to get container status \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": rpc error: code = NotFound desc = could not find container \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": container with ID starting with 0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34 not found: ID does not exist" Nov 03 19:32:52.959724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959422 4314 scope.go:122] "RemoveContainer" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.959724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959640 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} err="failed to get container status \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": rpc error: code = NotFound desc = could not find container \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": container with ID starting with d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918 not found: ID does not exist" Nov 03 19:32:52.959724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959656 4314 scope.go:122] "RemoveContainer" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.959906 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959839 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} err="failed to get container status \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": rpc error: code = NotFound desc = could not find container \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": container with ID starting with 4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723 not found: ID does not exist" Nov 03 19:32:52.959906 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.959854 4314 scope.go:122] "RemoveContainer" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.960076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960062 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} err="failed to get container status \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": rpc error: code = NotFound desc = could not find container \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": container with ID starting with 0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088 not found: ID does not exist" Nov 03 19:32:52.960113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960075 4314 scope.go:122] "RemoveContainer" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.960294 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960277 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} err="failed to get container status \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": rpc error: code = NotFound desc = could not find container \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": container with ID starting with 85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570 not found: ID does not exist" Nov 03 19:32:52.960341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960294 4314 scope.go:122] "RemoveContainer" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.960485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960465 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} err="failed to get container status \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": rpc error: code = NotFound desc = could not find container \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": container with ID starting with c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec not found: ID does not exist" Nov 03 19:32:52.960522 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960485 4314 scope.go:122] "RemoveContainer" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.960694 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960677 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} err="failed to get container status \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": rpc error: code = NotFound desc = could not find container \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": container with ID starting with 5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc not found: ID does not exist" Nov 03 19:32:52.960733 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960694 4314 scope.go:122] "RemoveContainer" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.960859 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960842 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34"} err="failed to get container status \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": rpc error: code = NotFound desc = could not find container \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": container with ID starting with 0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34 not found: ID does not exist" Nov 03 19:32:52.960859 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.960859 4314 scope.go:122] "RemoveContainer" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.961021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961004 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} err="failed to get container status \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": rpc error: code = NotFound desc = could not find container \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": container with ID starting with d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918 not found: ID does not exist" Nov 03 19:32:52.961076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961021 4314 scope.go:122] "RemoveContainer" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.961218 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961201 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} err="failed to get container status \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": rpc error: code = NotFound desc = could not find container \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": container with ID starting with 4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723 not found: ID does not exist" Nov 03 19:32:52.961291 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961218 4314 scope.go:122] "RemoveContainer" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.961429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961411 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} err="failed to get container status \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": rpc error: code = NotFound desc = could not find container \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": container with ID starting with 0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088 not found: ID does not exist" Nov 03 19:32:52.961473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961430 4314 scope.go:122] "RemoveContainer" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.961660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961642 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} err="failed to get container status \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": rpc error: code = NotFound desc = could not find container \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": container with ID starting with 85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570 not found: ID does not exist" Nov 03 19:32:52.961713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961661 4314 scope.go:122] "RemoveContainer" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.961883 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961870 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} err="failed to get container status \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": rpc error: code = NotFound desc = could not find container \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": container with ID starting with c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec not found: ID does not exist" Nov 03 19:32:52.961921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.961884 4314 scope.go:122] "RemoveContainer" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.962079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962061 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} err="failed to get container status \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": rpc error: code = NotFound desc = could not find container \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": container with ID starting with 5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc not found: ID does not exist" Nov 03 19:32:52.962079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962076 4314 scope.go:122] "RemoveContainer" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.962299 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962279 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34"} err="failed to get container status \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": rpc error: code = NotFound desc = could not find container \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": container with ID starting with 0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34 not found: ID does not exist" Nov 03 19:32:52.962363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962300 4314 scope.go:122] "RemoveContainer" containerID="d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918" Nov 03 19:32:52.962472 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962454 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918"} err="failed to get container status \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": rpc error: code = NotFound desc = could not find container \"d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918\": container with ID starting with d48fd8142c2c2e502a0ebc8b48e1659a4433b2c2c4fca4bff83bab2d026d2918 not found: ID does not exist" Nov 03 19:32:52.962509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962474 4314 scope.go:122] "RemoveContainer" containerID="4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723" Nov 03 19:32:52.962649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962627 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:52.962738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962713 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723"} err="failed to get container status \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": rpc error: code = NotFound desc = could not find container \"4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723\": container with ID starting with 4bdae34273dd889849bbd3190a67c8f25607472e99fe4faa99da8aa8c8bf8723 not found: ID does not exist" Nov 03 19:32:52.962738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962729 4314 scope.go:122] "RemoveContainer" containerID="0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088" Nov 03 19:32:52.962968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962941 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088"} err="failed to get container status \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": rpc error: code = NotFound desc = could not find container \"0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088\": container with ID starting with 0d2cc73c12a01afc0f3d746e0317cc1251fbee983612f867e8a46b28387c3088 not found: ID does not exist" Nov 03 19:32:52.963038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.962969 4314 scope.go:122] "RemoveContainer" containerID="85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570" Nov 03 19:32:52.963211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.963188 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570"} err="failed to get container status \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": rpc error: code = NotFound desc = could not find container \"85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570\": container with ID starting with 85a4220455417b4825167e7841236c32cdddcde2abc2411bb7c7fc4403d6d570 not found: ID does not exist" Nov 03 19:32:52.963211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.963212 4314 scope.go:122] "RemoveContainer" containerID="c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec" Nov 03 19:32:52.963464 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.963444 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec"} err="failed to get container status \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": rpc error: code = NotFound desc = could not find container \"c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec\": container with ID starting with c8114b68c51885065c6ee2b7557385eb3446ef67328fc5e5ddb0d17e27b31bec not found: ID does not exist" Nov 03 19:32:52.963518 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.963466 4314 scope.go:122] "RemoveContainer" containerID="5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc" Nov 03 19:32:52.963730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.963711 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc"} err="failed to get container status \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": rpc error: code = NotFound desc = could not find container \"5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc\": container with ID starting with 5ad8548071f8657f8ce8b714d977313f825252d1e50a12f0ec6184d2421969cc not found: ID does not exist" Nov 03 19:32:52.963730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.963729 4314 scope.go:122] "RemoveContainer" containerID="0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34" Nov 03 19:32:52.963952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.963934 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34"} err="failed to get container status \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": rpc error: code = NotFound desc = could not find container \"0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34\": container with ID starting with 0294684e6198cf274cb2a323fc7a93b35e1cc959539a4a691604fa837b2ebc34 not found: ID does not exist" Nov 03 19:32:52.966629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.966609 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:32:52.966757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.966744 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:32:52.966927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.966910 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 19:32:52.967125 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.967094 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:32:52.968913 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.968892 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:32:52.970140 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970104 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:32:52.970140 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970117 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:32:52.970269 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970233 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:32:52.970434 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970418 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:32:52.970506 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970464 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:32:52.970506 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970495 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:32:52.970661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970518 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ahgouu4ist57l\"" Nov 03 19:32:52.970661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.970618 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w2tdz\"" Nov 03 19:32:52.971788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.971773 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:32:52.972041 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.972023 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:32:52.975351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.975329 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:32:52.975351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.975338 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:32:52.976478 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:52.976456 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:32:53.097006 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.096970 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e86509fa-ee71-4083-8943-491c49eebcfe-config-out\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097006 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097008 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097034 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097077 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097165 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097208 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097231 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nwg\" (UniqueName: \"kubernetes.io/projected/e86509fa-ee71-4083-8943-491c49eebcfe-kube-api-access-l2nwg\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097273 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097340 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097363 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-web-config\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097384 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097408 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e86509fa-ee71-4083-8943-491c49eebcfe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097447 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097470 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097517 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097619 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097643 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097667 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097704 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.097982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.097763 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-config\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.198750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198678 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-config\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.198750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198714 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e86509fa-ee71-4083-8943-491c49eebcfe-config-out\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.198750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198740 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.198987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198768 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.198987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198790 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.198987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198822 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.198987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198856 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.198984 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nwg\" (UniqueName: \"kubernetes.io/projected/e86509fa-ee71-4083-8943-491c49eebcfe-kube-api-access-l2nwg\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199025 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199061 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199094 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-web-config\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199130 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199160 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e86509fa-ee71-4083-8943-491c49eebcfe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199183 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199212 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199243 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199293 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199318 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199349 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199378 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199840 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199622 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199840 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199644 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199840 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199662 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.199992 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.199970 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.200044 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.200013 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.200292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.200185 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.201755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.201725 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-config\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.202138 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.202114 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e86509fa-ee71-4083-8943-491c49eebcfe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.202262 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.202237 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e86509fa-ee71-4083-8943-491c49eebcfe-config-out\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.203166 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.202653 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.203166 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.202833 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-web-config\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.203166 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.203043 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.203166 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.203115 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.203422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.203392 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.203772 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.203749 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e86509fa-ee71-4083-8943-491c49eebcfe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.204238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.204215 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.204602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.204564 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.204700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.204650 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.204852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.204835 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e86509fa-ee71-4083-8943-491c49eebcfe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.211069 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.211052 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nwg\" (UniqueName: \"kubernetes.io/projected/e86509fa-ee71-4083-8943-491c49eebcfe-kube-api-access-l2nwg\") pod \"prometheus-k8s-0\" (UID: \"e86509fa-ee71-4083-8943-491c49eebcfe\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.282985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.282955 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:32:53.569838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.568767 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:32:53.571888 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:32:53.571856 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86509fa_ee71_4083_8943_491c49eebcfe.slice/crio-dca166ca33c3f5ec05cf877ab20269c08cfe4fe3618f2c63417429abe29b7486 WatchSource:0}: Error finding container dca166ca33c3f5ec05cf877ab20269c08cfe4fe3618f2c63417429abe29b7486: Status 404 returned error can't find the container with id dca166ca33c3f5ec05cf877ab20269c08cfe4fe3618f2c63417429abe29b7486 Nov 03 19:32:53.910521 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.910437 4314 generic.go:356] "Generic (PLEG): container finished" podID="e86509fa-ee71-4083-8943-491c49eebcfe" containerID="d5c0e30c2e5b996a4084c1e8dc523b92db0869024574e190cc204e25248ce2ef" exitCode=0 Nov 03 19:32:53.910707 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.910530 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerDied","Data":"d5c0e30c2e5b996a4084c1e8dc523b92db0869024574e190cc204e25248ce2ef"} Nov 03 19:32:53.910707 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:53.910635 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerStarted","Data":"dca166ca33c3f5ec05cf877ab20269c08cfe4fe3618f2c63417429abe29b7486"} Nov 03 19:32:54.424774 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.424701 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb65768d-6e8c-4868-bd82-f3f5b149eeb2" path="/var/lib/kubelet/pods/fb65768d-6e8c-4868-bd82-f3f5b149eeb2/volumes" Nov 03 19:32:54.918858 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.918826 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerStarted","Data":"353c74eb8d929a492856c004d7f08a2d41356aa18812a3b27d6be730c8d5f46f"} Nov 03 19:32:54.918988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.918886 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerStarted","Data":"59a0b461985234c00a473c2606eb4596a9ebcb5f4d10cbef9118c81d7b1d037e"} Nov 03 19:32:54.918988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.918912 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerStarted","Data":"26d367097885b1c11225083e09b0cc7d56a7e9b35d864c8ab1e465892857948c"} Nov 03 19:32:54.918988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.918935 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerStarted","Data":"7f6b1d38b6c69bba76bd53c1ee60a72cfdd6c66ca0fe9fa0c4f8dc9c6901f704"} Nov 03 19:32:54.918988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.918960 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerStarted","Data":"b3de39b7b8fded9dc18e12045e86f10ec5bc2af0e84cab24f3a62c04b3835173"} Nov 03 19:32:54.918988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.918982 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e86509fa-ee71-4083-8943-491c49eebcfe","Type":"ContainerStarted","Data":"c98f4e3b4d2f304cf0edd882a382b520fb5deabdc51ec8e9742f7f04153ba6a9"} Nov 03 19:32:54.947682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:54.947633 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.927920481 podStartE2EDuration="2.947619685s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.699204ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:54.94525456 +0000 UTC m=+82.650093892" watchObservedRunningTime="2026-11-03 19:32:54.947619685 +0000 UTC m=+82.652459024" Nov 03 19:32:55.327074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:55.327046 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d799d7c5b-b55bb"] Nov 03 19:32:58.283622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:32:58.283566 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:33:20.347098 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.347057 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d799d7c5b-b55bb" podUID="935cd956-ac7a-426a-9cb3-20dc0c96582f" containerName="console" containerID="cri-o://a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c" gracePeriod=15 Nov 03 19:33:20.585967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.585938 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:33:20.600801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.600740 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-oauth-serving-cert\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.600801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.600785 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-serving-cert\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.600935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.600807 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935cd956-ac7a-426a-9cb3-20dc0c96582f-kube-api-access-2cjtw\" (UniqueName: \"kubernetes.io/projected/935cd956-ac7a-426a-9cb3-20dc0c96582f-kube-api-access-2cjtw\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.600935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.600857 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-trusted-ca-bundle\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.600935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.600916 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935cd956-ac7a-426a-9cb3-20dc0c96582f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/935cd956-ac7a-426a-9cb3-20dc0c96582f-tmp\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.601081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.600950 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-oauth-config\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.601081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.600977 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-config\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-config\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.601251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601102 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-oauth-serving-cert" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:20.601251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601231 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935cd956-ac7a-426a-9cb3-20dc0c96582f-tmp" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:33:20.601363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601314 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-config" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:20.601363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601342 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-service-ca\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-service-ca\") pod \"935cd956-ac7a-426a-9cb3-20dc0c96582f\" (UID: \"935cd956-ac7a-426a-9cb3-20dc0c96582f\") " Nov 03 19:33:20.601363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601342 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-trusted-ca-bundle" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:20.601702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601661 4314 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-trusted-ca-bundle\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:20.601796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601710 4314 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935cd956-ac7a-426a-9cb3-20dc0c96582f-tmp\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:20.601796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601726 4314 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:20.601796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601741 4314 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-oauth-serving-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:20.601796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.601759 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-service-ca" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:33:20.603977 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.603783 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-oauth-config" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:20.603977 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.603848 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-serving-cert" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:33:20.604278 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.604254 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935cd956-ac7a-426a-9cb3-20dc0c96582f-kube-api-access-2cjtw" pod "935cd956-ac7a-426a-9cb3-20dc0c96582f" (UID: "935cd956-ac7a-426a-9cb3-20dc0c96582f"). InnerVolumeSpecName "kube-api-access-2cjtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:33:20.702790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.702755 4314 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-oauth-config\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:20.702790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.702787 4314 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935cd956-ac7a-426a-9cb3-20dc0c96582f-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:20.702790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.702798 4314 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935cd956-ac7a-426a-9cb3-20dc0c96582f-console-serving-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:20.702978 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:20.702807 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cjtw\" (UniqueName: \"kubernetes.io/projected/935cd956-ac7a-426a-9cb3-20dc0c96582f-kube-api-access-2cjtw\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:33:21.013881 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.013854 4314 generic.go:356] "Generic (PLEG): container finished" podID="935cd956-ac7a-426a-9cb3-20dc0c96582f" containerID="a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c" exitCode=0 Nov 03 19:33:21.014015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.013899 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799d7c5b-b55bb" event={"ID":"935cd956-ac7a-426a-9cb3-20dc0c96582f","Type":"ContainerDied","Data":"a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c"} Nov 03 19:33:21.014015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.013968 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d799d7c5b-b55bb" event={"ID":"935cd956-ac7a-426a-9cb3-20dc0c96582f","Type":"ContainerDied","Data":"2905c1edb99fc0413bd6c4c811529a6180a13e44c9092cdce8d23fec73dddeef"} Nov 03 19:33:21.014015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.013901 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d799d7c5b-b55bb" Nov 03 19:33:21.014110 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.013987 4314 scope.go:122] "RemoveContainer" containerID="a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c" Nov 03 19:33:21.022327 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.022311 4314 scope.go:122] "RemoveContainer" containerID="a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c" Nov 03 19:33:21.022605 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:33:21.022565 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c\": container with ID starting with a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c not found: ID does not exist" containerID="a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c" Nov 03 19:33:21.022684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.022608 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c"} err="failed to get container status \"a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c\": rpc error: code = NotFound desc = could not find container \"a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c\": container with ID starting with a2d8bc00ebdb9904e21e2a9ff065cba2da7a312a547051964fe251372a205b5c not found: ID does not exist" Nov 03 19:33:21.034267 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.034245 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d799d7c5b-b55bb"] Nov 03 19:33:21.037225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:21.037207 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d799d7c5b-b55bb"] Nov 03 19:33:21.988179 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:33:21.988136 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:33:22.424336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:22.424304 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935cd956-ac7a-426a-9cb3-20dc0c96582f" path="/var/lib/kubelet/pods/935cd956-ac7a-426a-9cb3-20dc0c96582f/volumes" Nov 03 19:33:51.988499 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:33:51.988409 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:33:53.283533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:53.283502 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:33:53.298643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:53.298619 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:33:54.139612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:33:54.139567 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:34:21.988751 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:34:21.988713 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:34:51.987434 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:34:51.987399 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:35:21.988411 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:35:21.988377 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:35:51.987931 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:35:51.987886 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:36:21.988678 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:36:21.988639 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:36:23.723717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.723684 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6"] Nov 03 19:36:23.724176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.723954 4314 cpu_manager.go:414] "containerMap: removing container" podUID="935cd956-ac7a-426a-9cb3-20dc0c96582f" containerName="console" Nov 03 19:36:23.724176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.723964 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935cd956-ac7a-426a-9cb3-20dc0c96582f" containerName="console" Nov 03 19:36:23.724176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.724032 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="935cd956-ac7a-426a-9cb3-20dc0c96582f" containerName="console" Nov 03 19:36:23.726826 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.726806 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:23.735490 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.735471 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 19:36:23.735646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.735611 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-29xhm\"" Nov 03 19:36:23.735705 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.735654 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:36:23.736621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.736602 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6"] Nov 03 19:36:23.806722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.806699 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f-tmp\") pod \"jobset-operator-674cd59dc5-szdd6\" (UID: \"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:23.806822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.806768 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fw4\" (UniqueName: \"kubernetes.io/projected/bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f-kube-api-access-z2fw4\") pod \"jobset-operator-674cd59dc5-szdd6\" (UID: \"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:23.907853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.907830 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fw4\" (UniqueName: \"kubernetes.io/projected/bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f-kube-api-access-z2fw4\") pod \"jobset-operator-674cd59dc5-szdd6\" (UID: \"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:23.907932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.907860 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f-tmp\") pod \"jobset-operator-674cd59dc5-szdd6\" (UID: \"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:23.908171 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.908157 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f-tmp\") pod \"jobset-operator-674cd59dc5-szdd6\" (UID: \"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:23.915775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:23.915754 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fw4\" (UniqueName: \"kubernetes.io/projected/bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f-kube-api-access-z2fw4\") pod \"jobset-operator-674cd59dc5-szdd6\" (UID: \"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:24.047358 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:24.047339 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" Nov 03 19:36:24.319684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:24.319611 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6"] Nov 03 19:36:24.322874 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:36:24.322845 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd09c84b_16b6_4b4d_8a6f_ad92d9a1233f.slice/crio-efdaaffc982d436d30a3d1adb4bb7d60ef2509db208193935595f169958e1091 WatchSource:0}: Error finding container efdaaffc982d436d30a3d1adb4bb7d60ef2509db208193935595f169958e1091: Status 404 returned error can't find the container with id efdaaffc982d436d30a3d1adb4bb7d60ef2509db208193935595f169958e1091 Nov 03 19:36:24.608006 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:24.607928 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" event={"ID":"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f","Type":"ContainerStarted","Data":"efdaaffc982d436d30a3d1adb4bb7d60ef2509db208193935595f169958e1091"} Nov 03 19:36:28.623599 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:28.623543 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" event={"ID":"bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f","Type":"ContainerStarted","Data":"54fb080d29f81796619d72b180f7e1d6f0f3836be07a0c8a1c404b3646d236c8"} Nov 03 19:36:28.640307 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:28.640249 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-szdd6" podStartSLOduration=2.063669861 podStartE2EDuration="5.64023276s" totalImagesPullingTime="3.576562899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:28.639686225 +0000 UTC m=+296.344525579" watchObservedRunningTime="2026-11-03 19:36:28.64023276 +0000 UTC m=+296.345072102" Nov 03 19:36:32.438743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:32.438712 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:36:32.439182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:32.438781 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:36:34.828302 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.828266 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5"] Nov 03 19:36:34.832146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.832123 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.858492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.858467 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5"] Nov 03 19:36:34.890481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.890459 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0989ad6-060e-4da2-a646-6095e895ad09-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.890617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.890501 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.890673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.890625 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggzl\" (UniqueName: \"kubernetes.io/projected/e0989ad6-060e-4da2-a646-6095e895ad09-kube-api-access-9ggzl\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.890673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.890659 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.890780 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.890749 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0989ad6-060e-4da2-a646-6095e895ad09-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.890831 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.890787 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.921982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.921962 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 19:36:34.922222 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.922203 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 19:36:34.922299 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.922262 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 19:36:34.922299 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.922276 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 19:36:34.922299 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.922278 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 19:36:34.922417 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.922339 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dpmf9\"" Nov 03 19:36:34.991187 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.991164 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0989ad6-060e-4da2-a646-6095e895ad09-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.991296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.991202 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.991296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.991252 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggzl\" (UniqueName: \"kubernetes.io/projected/e0989ad6-060e-4da2-a646-6095e895ad09-kube-api-access-9ggzl\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.991414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.991293 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.991414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.991327 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0989ad6-060e-4da2-a646-6095e895ad09-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.991414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.991353 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.991695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.991670 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0989ad6-060e-4da2-a646-6095e895ad09-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.993828 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.993804 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0989ad6-060e-4da2-a646-6095e895ad09-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.993914 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.993847 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.993983 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.993969 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:34.994088 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:34.994068 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0989ad6-060e-4da2-a646-6095e895ad09-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:35.001635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:35.001612 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggzl\" (UniqueName: \"kubernetes.io/projected/e0989ad6-060e-4da2-a646-6095e895ad09-kube-api-access-9ggzl\") pod \"opendatahub-operator-controller-manager-bd9746b54-5blw5\" (UID: \"e0989ad6-060e-4da2-a646-6095e895ad09\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:35.146408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:35.146331 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:35.441603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:35.441512 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5"] Nov 03 19:36:35.445250 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:36:35.445220 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0989ad6_060e_4da2_a646_6095e895ad09.slice/crio-6cd66e07954f698089e9cf68d3b58f455b5a5226f2600f49a283878025e23f80 WatchSource:0}: Error finding container 6cd66e07954f698089e9cf68d3b58f455b5a5226f2600f49a283878025e23f80: Status 404 returned error can't find the container with id 6cd66e07954f698089e9cf68d3b58f455b5a5226f2600f49a283878025e23f80 Nov 03 19:36:35.447106 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:35.447080 4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:36:35.652889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:35.652857 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" event={"ID":"e0989ad6-060e-4da2-a646-6095e895ad09","Type":"ContainerStarted","Data":"6cd66e07954f698089e9cf68d3b58f455b5a5226f2600f49a283878025e23f80"} Nov 03 19:36:38.666902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:38.666870 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" event={"ID":"e0989ad6-060e-4da2-a646-6095e895ad09","Type":"ContainerStarted","Data":"1d1475c2aa193c4fb68e514a3b7afbdd96588796aa78f11a9c1d6fa87c82fa67"} Nov 03 19:36:39.672461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:39.672426 4314 generic.go:356] "Generic (PLEG): container finished" podID="e0989ad6-060e-4da2-a646-6095e895ad09" containerID="1d1475c2aa193c4fb68e514a3b7afbdd96588796aa78f11a9c1d6fa87c82fa67" exitCode=0 Nov 03 19:36:39.672929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:39.672480 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" event={"ID":"e0989ad6-060e-4da2-a646-6095e895ad09","Type":"ContainerDied","Data":"1d1475c2aa193c4fb68e514a3b7afbdd96588796aa78f11a9c1d6fa87c82fa67"} Nov 03 19:36:40.656684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.656652 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52"] Nov 03 19:36:40.659826 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.659810 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:40.671051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.671030 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 19:36:40.671189 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.671055 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 19:36:40.675261 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.675242 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9spdc\"" Nov 03 19:36:40.677842 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.677815 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" event={"ID":"e0989ad6-060e-4da2-a646-6095e895ad09","Type":"ContainerStarted","Data":"4d76a684f4fc2169e348b8531266aaaf3a918ab72d1170adda3501ee04edc996"} Nov 03 19:36:40.677971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.677953 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:40.694965 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.694940 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52"] Nov 03 19:36:40.738168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.738142 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b56fa10-9fe9-43a5-b0f2-b6a3277188a5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vlp52\" (UID: \"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:40.738304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.738273 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cggj\" (UniqueName: \"kubernetes.io/projected/8b56fa10-9fe9-43a5-b0f2-b6a3277188a5-kube-api-access-5cggj\") pod \"cert-manager-cainjector-5c57bbd4fc-vlp52\" (UID: \"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:40.740504 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.740462 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" podStartSLOduration=2.94801148 podStartE2EDuration="6.740446672s" totalImagesPullingTime="3.227961604s" totalInitContainerRuntime="564.473588ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:40.73785693 +0000 UTC m=+308.442696270" watchObservedRunningTime="2026-11-03 19:36:40.740446672 +0000 UTC m=+308.445286012" Nov 03 19:36:40.839163 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.839136 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cggj\" (UniqueName: \"kubernetes.io/projected/8b56fa10-9fe9-43a5-b0f2-b6a3277188a5-kube-api-access-5cggj\") pod \"cert-manager-cainjector-5c57bbd4fc-vlp52\" (UID: \"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:40.839271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.839183 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b56fa10-9fe9-43a5-b0f2-b6a3277188a5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vlp52\" (UID: \"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:40.850515 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.850490 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cggj\" (UniqueName: \"kubernetes.io/projected/8b56fa10-9fe9-43a5-b0f2-b6a3277188a5-kube-api-access-5cggj\") pod \"cert-manager-cainjector-5c57bbd4fc-vlp52\" (UID: \"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:40.850872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.850850 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b56fa10-9fe9-43a5-b0f2-b6a3277188a5-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-vlp52\" (UID: \"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:40.970226 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:40.970171 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" Nov 03 19:36:41.244493 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:41.244463 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52"] Nov 03 19:36:41.247830 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:36:41.247799 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b56fa10_9fe9_43a5_b0f2_b6a3277188a5.slice/crio-ca2bcfffaceb55781248bb50b40164c5bf09e522e6427094315ff99d652a5605 WatchSource:0}: Error finding container ca2bcfffaceb55781248bb50b40164c5bf09e522e6427094315ff99d652a5605: Status 404 returned error can't find the container with id ca2bcfffaceb55781248bb50b40164c5bf09e522e6427094315ff99d652a5605 Nov 03 19:36:41.684844 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:41.684759 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" event={"ID":"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5","Type":"ContainerStarted","Data":"ca2bcfffaceb55781248bb50b40164c5bf09e522e6427094315ff99d652a5605"} Nov 03 19:36:44.698334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:44.698303 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" event={"ID":"8b56fa10-9fe9-43a5-b0f2-b6a3277188a5","Type":"ContainerStarted","Data":"a8b98998035d80d422f7e713dea94e4121dd42747f0b7533c7862c201596ef84"} Nov 03 19:36:44.716077 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:44.716036 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-vlp52" podStartSLOduration=1.323969828 podStartE2EDuration="4.716022727s" totalImagesPullingTime="3.392052899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:44.715869892 +0000 UTC m=+312.420709253" watchObservedRunningTime="2026-11-03 19:36:44.716022727 +0000 UTC m=+312.420862064" Nov 03 19:36:50.655633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.655598 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hnm2n"] Nov 03 19:36:50.719203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.719169 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hnm2n"] Nov 03 19:36:50.719357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.719277 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:50.723949 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.723922 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-66dbv\"" Nov 03 19:36:50.824005 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.823977 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5ll\" (UniqueName: \"kubernetes.io/projected/7cba88b9-583b-44ed-af31-2a1ea13e939e-kube-api-access-tw5ll\") pod \"cert-manager-597b95cb84-hnm2n\" (UID: \"7cba88b9-583b-44ed-af31-2a1ea13e939e\") " pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:50.824118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.824021 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cba88b9-583b-44ed-af31-2a1ea13e939e-bound-sa-token\") pod \"cert-manager-597b95cb84-hnm2n\" (UID: \"7cba88b9-583b-44ed-af31-2a1ea13e939e\") " pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:50.924567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.924509 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5ll\" (UniqueName: \"kubernetes.io/projected/7cba88b9-583b-44ed-af31-2a1ea13e939e-kube-api-access-tw5ll\") pod \"cert-manager-597b95cb84-hnm2n\" (UID: \"7cba88b9-583b-44ed-af31-2a1ea13e939e\") " pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:50.924567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.924554 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cba88b9-583b-44ed-af31-2a1ea13e939e-bound-sa-token\") pod \"cert-manager-597b95cb84-hnm2n\" (UID: \"7cba88b9-583b-44ed-af31-2a1ea13e939e\") " pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:50.932654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.932634 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7cba88b9-583b-44ed-af31-2a1ea13e939e-bound-sa-token\") pod \"cert-manager-597b95cb84-hnm2n\" (UID: \"7cba88b9-583b-44ed-af31-2a1ea13e939e\") " pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:50.932756 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:50.932712 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5ll\" (UniqueName: \"kubernetes.io/projected/7cba88b9-583b-44ed-af31-2a1ea13e939e-kube-api-access-tw5ll\") pod \"cert-manager-597b95cb84-hnm2n\" (UID: \"7cba88b9-583b-44ed-af31-2a1ea13e939e\") " pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:51.041392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:51.041370 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hnm2n" Nov 03 19:36:51.334816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:51.334779 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hnm2n"] Nov 03 19:36:51.338083 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:36:51.338055 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cba88b9_583b_44ed_af31_2a1ea13e939e.slice/crio-b12ecd341cc8dbc1e48a35dc7a8af0af7d41e2fa6847ac324f0752c33b86fe0d WatchSource:0}: Error finding container b12ecd341cc8dbc1e48a35dc7a8af0af7d41e2fa6847ac324f0752c33b86fe0d: Status 404 returned error can't find the container with id b12ecd341cc8dbc1e48a35dc7a8af0af7d41e2fa6847ac324f0752c33b86fe0d Nov 03 19:36:51.686553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:51.686467 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-5blw5" Nov 03 19:36:51.729432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:51.729400 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hnm2n" event={"ID":"7cba88b9-583b-44ed-af31-2a1ea13e939e","Type":"ContainerStarted","Data":"3d8dbb1b351e1761e71b1ab99ecc163740b880c149fbdcc9aef215c00a4775dd"} Nov 03 19:36:51.729574 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:51.729442 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hnm2n" event={"ID":"7cba88b9-583b-44ed-af31-2a1ea13e939e","Type":"ContainerStarted","Data":"b12ecd341cc8dbc1e48a35dc7a8af0af7d41e2fa6847ac324f0752c33b86fe0d"} Nov 03 19:36:51.750146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:36:51.750086 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hnm2n" podStartSLOduration=1.750068663 podStartE2EDuration="1.750068663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:51.748765026 +0000 UTC m=+319.453604365" watchObservedRunningTime="2026-11-03 19:36:51.750068663 +0000 UTC m=+319.454908003" Nov 03 19:36:51.987605 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:36:51.987513 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:37:01.280111 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.280072 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm"] Nov 03 19:37:01.282894 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.282876 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.289279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.289253 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm"] Nov 03 19:37:01.290937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.290915 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 19:37:01.291919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.291901 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 19:37:01.292020 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.291925 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 19:37:01.292255 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.292225 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j2gnf\"" Nov 03 19:37:01.292255 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.292227 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 19:37:01.410343 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.410318 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59gt\" (UniqueName: \"kubernetes.io/projected/93fcd6a5-399a-4300-ad54-afdd837f808a-kube-api-access-h59gt\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.410479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.410359 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93fcd6a5-399a-4300-ad54-afdd837f808a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.410479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.410392 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/93fcd6a5-399a-4300-ad54-afdd837f808a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.510805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.510756 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59gt\" (UniqueName: \"kubernetes.io/projected/93fcd6a5-399a-4300-ad54-afdd837f808a-kube-api-access-h59gt\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.510989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.510833 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93fcd6a5-399a-4300-ad54-afdd837f808a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.510989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.510884 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/93fcd6a5-399a-4300-ad54-afdd837f808a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.511523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.511499 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/93fcd6a5-399a-4300-ad54-afdd837f808a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.513257 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.513235 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/93fcd6a5-399a-4300-ad54-afdd837f808a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.519498 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.519477 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59gt\" (UniqueName: \"kubernetes.io/projected/93fcd6a5-399a-4300-ad54-afdd837f808a-kube-api-access-h59gt\") pod \"kuadrant-console-plugin-66c4fb575d-ffrlm\" (UID: \"93fcd6a5-399a-4300-ad54-afdd837f808a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.594834 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.594776 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" Nov 03 19:37:01.862684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:01.862470 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm"] Nov 03 19:37:01.865130 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:37:01.865097 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93fcd6a5_399a_4300_ad54_afdd837f808a.slice/crio-1f4d265d117aecc08b6d1b4bbd09c78d17b9c51ea67147c3b1797c017595b1bd WatchSource:0}: Error finding container 1f4d265d117aecc08b6d1b4bbd09c78d17b9c51ea67147c3b1797c017595b1bd: Status 404 returned error can't find the container with id 1f4d265d117aecc08b6d1b4bbd09c78d17b9c51ea67147c3b1797c017595b1bd Nov 03 19:37:02.776879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:02.776845 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" event={"ID":"93fcd6a5-399a-4300-ad54-afdd837f808a","Type":"ContainerStarted","Data":"1f4d265d117aecc08b6d1b4bbd09c78d17b9c51ea67147c3b1797c017595b1bd"} Nov 03 19:37:07.802035 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:07.802001 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" event={"ID":"93fcd6a5-399a-4300-ad54-afdd837f808a","Type":"ContainerStarted","Data":"14ad9516946332dd6d3fdcaf837ba50aa962293045416fbe84fdfc8fd9f6ac8e"} Nov 03 19:37:07.817924 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:07.817879 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ffrlm" podStartSLOduration=1.803989676 podStartE2EDuration="6.817864852s" totalImagesPullingTime="5.013875176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:07.817501633 +0000 UTC m=+335.522340973" watchObservedRunningTime="2026-11-03 19:37:07.817864852 +0000 UTC m=+335.522704191" Nov 03 19:37:08.832021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.831988 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f46798c78-dmhc8"] Nov 03 19:37:08.835346 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.835327 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:08.844698 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.844669 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 19:37:08.844809 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.844783 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 19:37:08.844809 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.844803 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 19:37:08.844884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.844832 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r6qwz\"" Nov 03 19:37:08.844884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.844870 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 19:37:08.848693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.848667 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f46798c78-dmhc8"] Nov 03 19:37:08.971392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.971355 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee53b4-83e7-4974-807c-f89e3df209d9-tls-certs\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:08.971933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.971429 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzsqz\" (UniqueName: \"kubernetes.io/projected/f1ee53b4-83e7-4974-807c-f89e3df209d9-kube-api-access-fzsqz\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:08.971933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:08.971462 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1ee53b4-83e7-4974-807c-f89e3df209d9-tmp\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.072914 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.072880 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzsqz\" (UniqueName: \"kubernetes.io/projected/f1ee53b4-83e7-4974-807c-f89e3df209d9-kube-api-access-fzsqz\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.073102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.072932 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1ee53b4-83e7-4974-807c-f89e3df209d9-tmp\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.073102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.072995 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee53b4-83e7-4974-807c-f89e3df209d9-tls-certs\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.075254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.075226 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1ee53b4-83e7-4974-807c-f89e3df209d9-tmp\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.075535 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.075514 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee53b4-83e7-4974-807c-f89e3df209d9-tls-certs\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.080801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.080779 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzsqz\" (UniqueName: \"kubernetes.io/projected/f1ee53b4-83e7-4974-807c-f89e3df209d9-kube-api-access-fzsqz\") pod \"kube-auth-proxy-f46798c78-dmhc8\" (UID: \"f1ee53b4-83e7-4974-807c-f89e3df209d9\") " pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.147260 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.147204 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" Nov 03 19:37:09.418446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.418418 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f46798c78-dmhc8"] Nov 03 19:37:09.420925 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:37:09.420893 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ee53b4_83e7_4974_807c_f89e3df209d9.slice/crio-397f554c78504028c6eb08570362e378f2683dbfa5170408da6f8c57937cdb0b WatchSource:0}: Error finding container 397f554c78504028c6eb08570362e378f2683dbfa5170408da6f8c57937cdb0b: Status 404 returned error can't find the container with id 397f554c78504028c6eb08570362e378f2683dbfa5170408da6f8c57937cdb0b Nov 03 19:37:09.811590 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:09.811556 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" event={"ID":"f1ee53b4-83e7-4974-807c-f89e3df209d9","Type":"ContainerStarted","Data":"397f554c78504028c6eb08570362e378f2683dbfa5170408da6f8c57937cdb0b"} Nov 03 19:37:10.108198 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.108122 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd"] Nov 03 19:37:10.111524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.111503 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.115822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.115799 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z646t\"" Nov 03 19:37:10.125433 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.125409 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd"] Nov 03 19:37:10.285572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.285529 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9545\" (UniqueName: \"kubernetes.io/projected/8cd08483-1bd3-46f7-a415-47c5fc688cf8-kube-api-access-v9545\") pod \"kuadrant-operator-controller-manager-678d87766f-8bntd\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.285765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.285696 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cd08483-1bd3-46f7-a415-47c5fc688cf8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8bntd\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.387179 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.387098 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9545\" (UniqueName: \"kubernetes.io/projected/8cd08483-1bd3-46f7-a415-47c5fc688cf8-kube-api-access-v9545\") pod \"kuadrant-operator-controller-manager-678d87766f-8bntd\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.387179 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.387165 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cd08483-1bd3-46f7-a415-47c5fc688cf8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8bntd\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.387622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.387593 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cd08483-1bd3-46f7-a415-47c5fc688cf8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8bntd\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.396882 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.396856 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9545\" (UniqueName: \"kubernetes.io/projected/8cd08483-1bd3-46f7-a415-47c5fc688cf8-kube-api-access-v9545\") pod \"kuadrant-operator-controller-manager-678d87766f-8bntd\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.424620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.424599 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:10.704295 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.704210 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd"] Nov 03 19:37:10.713317 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:37:10.713277 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd08483_1bd3_46f7_a415_47c5fc688cf8.slice/crio-dbfba45e7fe8e2aa3aceb115f61b071a7e6c0f9cf780dee1af99d64b69e5b556 WatchSource:0}: Error finding container dbfba45e7fe8e2aa3aceb115f61b071a7e6c0f9cf780dee1af99d64b69e5b556: Status 404 returned error can't find the container with id dbfba45e7fe8e2aa3aceb115f61b071a7e6c0f9cf780dee1af99d64b69e5b556 Nov 03 19:37:10.819827 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:10.819786 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" event={"ID":"8cd08483-1bd3-46f7-a415-47c5fc688cf8","Type":"ContainerStarted","Data":"dbfba45e7fe8e2aa3aceb115f61b071a7e6c0f9cf780dee1af99d64b69e5b556"} Nov 03 19:37:12.838217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:12.838172 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" event={"ID":"f1ee53b4-83e7-4974-807c-f89e3df209d9","Type":"ContainerStarted","Data":"7772ee57cfaee171a2edd3e3243755be6f79d700d9d203cf686e3d6af2f52c3c"} Nov 03 19:37:12.856353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:12.855265 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f46798c78-dmhc8" podStartSLOduration=1.9789108720000002 podStartE2EDuration="4.855249988s" totalImagesPullingTime="2.876339116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:12.854118643 +0000 UTC m=+340.558958006" watchObservedRunningTime="2026-11-03 19:37:12.855249988 +0000 UTC m=+340.560089329" Nov 03 19:37:15.853250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:15.853213 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" event={"ID":"8cd08483-1bd3-46f7-a415-47c5fc688cf8","Type":"ContainerStarted","Data":"90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d"} Nov 03 19:37:15.853704 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:15.853360 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:15.874846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:15.874796 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" podStartSLOduration=1.476347598 podStartE2EDuration="5.874782804s" totalImagesPullingTime="4.398435206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:15.872372555 +0000 UTC m=+343.577211896" watchObservedRunningTime="2026-11-03 19:37:15.874782804 +0000 UTC m=+343.579622143" Nov 03 19:37:21.991337 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:37:21.991300 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:37:26.859570 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:26.859539 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 19:37:43.878129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:43.878096 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:37:44.088952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.088911 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:37:44.088952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.088958 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:37:44.089148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.089034 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.152011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.151948 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 19:37:44.165822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.165797 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkgl\" (UniqueName: \"kubernetes.io/projected/81469cc9-29b4-4679-bc25-acacb15bd34d-kube-api-access-zjkgl\") pod \"limitador-limitador-7c9b87d574-n7khn\" (UID: \"81469cc9-29b4-4679-bc25-acacb15bd34d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.165932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.165847 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81469cc9-29b4-4679-bc25-acacb15bd34d-config-file\") pod \"limitador-limitador-7c9b87d574-n7khn\" (UID: \"81469cc9-29b4-4679-bc25-acacb15bd34d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.266712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.266682 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81469cc9-29b4-4679-bc25-acacb15bd34d-config-file\") pod \"limitador-limitador-7c9b87d574-n7khn\" (UID: \"81469cc9-29b4-4679-bc25-acacb15bd34d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.266852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.266748 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkgl\" (UniqueName: \"kubernetes.io/projected/81469cc9-29b4-4679-bc25-acacb15bd34d-kube-api-access-zjkgl\") pod \"limitador-limitador-7c9b87d574-n7khn\" (UID: \"81469cc9-29b4-4679-bc25-acacb15bd34d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.267290 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.267270 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81469cc9-29b4-4679-bc25-acacb15bd34d-config-file\") pod \"limitador-limitador-7c9b87d574-n7khn\" (UID: \"81469cc9-29b4-4679-bc25-acacb15bd34d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.274976 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.274950 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkgl\" (UniqueName: \"kubernetes.io/projected/81469cc9-29b4-4679-bc25-acacb15bd34d-kube-api-access-zjkgl\") pod \"limitador-limitador-7c9b87d574-n7khn\" (UID: \"81469cc9-29b4-4679-bc25-acacb15bd34d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.410996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.410919 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:44.705040 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.704955 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:37:44.708572 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:37:44.708542 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81469cc9_29b4_4679_bc25_acacb15bd34d.slice/crio-3c6e78782ae5793693ad92382e4123fb9ec66c65ecb879a9f9d8ad2280e0857e WatchSource:0}: Error finding container 3c6e78782ae5793693ad92382e4123fb9ec66c65ecb879a9f9d8ad2280e0857e: Status 404 returned error can't find the container with id 3c6e78782ae5793693ad92382e4123fb9ec66c65ecb879a9f9d8ad2280e0857e Nov 03 19:37:44.965791 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:44.965706 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" event={"ID":"81469cc9-29b4-4679-bc25-acacb15bd34d","Type":"ContainerStarted","Data":"3c6e78782ae5793693ad92382e4123fb9ec66c65ecb879a9f9d8ad2280e0857e"} Nov 03 19:37:47.978563 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:47.978522 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" event={"ID":"81469cc9-29b4-4679-bc25-acacb15bd34d","Type":"ContainerStarted","Data":"8729a97c60ce0f925777497dd2a295dfd227681767967555fc9cc3b89b5ed198"} Nov 03 19:37:47.978974 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:47.978716 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:37:47.994482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:47.994433 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" podStartSLOduration=2.714828529 podStartE2EDuration="4.994420848s" totalImagesPullingTime="2.279592319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:47.993527143 +0000 UTC m=+375.698366483" watchObservedRunningTime="2026-11-03 19:37:47.994420848 +0000 UTC m=+375.699260188" Nov 03 19:37:51.987309 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:37:51.987271 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:37:58.983691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:37:58.983661 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-n7khn" Nov 03 19:38:05.505137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.505105 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-xp8mn"] Nov 03 19:38:05.507659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.507641 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.511649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.511628 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 19:38:05.516782 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.516760 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xp8mn"] Nov 03 19:38:05.559831 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.559809 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 19:38:05.559939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.559858 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 19:38:05.559939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.559882 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9b49k\"" Nov 03 19:38:05.612010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.611991 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 19:38:05.655921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.655900 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nm85\" (UniqueName: \"kubernetes.io/projected/89d72820-7b45-41bd-8252-b25c4c1ef048-kube-api-access-7nm85\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.656021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.655930 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89d72820-7b45-41bd-8252-b25c4c1ef048-cert\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.656021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.655955 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d72820-7b45-41bd-8252-b25c4c1ef048-metrics-certs\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.756855 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.756802 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nm85\" (UniqueName: \"kubernetes.io/projected/89d72820-7b45-41bd-8252-b25c4c1ef048-kube-api-access-7nm85\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.756855 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.756834 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89d72820-7b45-41bd-8252-b25c4c1ef048-cert\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.756855 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.756854 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d72820-7b45-41bd-8252-b25c4c1ef048-metrics-certs\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.759175 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.759155 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89d72820-7b45-41bd-8252-b25c4c1ef048-cert\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.759252 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.759186 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89d72820-7b45-41bd-8252-b25c4c1ef048-metrics-certs\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.767325 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.767303 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nm85\" (UniqueName: \"kubernetes.io/projected/89d72820-7b45-41bd-8252-b25c4c1ef048-kube-api-access-7nm85\") pod \"odh-model-controller-769875db56-xp8mn\" (UID: \"89d72820-7b45-41bd-8252-b25c4c1ef048\") " pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:05.820999 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:05.820978 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:06.091255 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:06.091183 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-xp8mn"] Nov 03 19:38:06.097956 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:38:06.097924 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d72820_7b45_41bd_8252_b25c4c1ef048.slice/crio-c041382b85d6812dcd5bdf1b5ae9da982971376e60cf6eb6ee589406503a7734 WatchSource:0}: Error finding container c041382b85d6812dcd5bdf1b5ae9da982971376e60cf6eb6ee589406503a7734: Status 404 returned error can't find the container with id c041382b85d6812dcd5bdf1b5ae9da982971376e60cf6eb6ee589406503a7734 Nov 03 19:38:07.060320 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:07.060270 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xp8mn" event={"ID":"89d72820-7b45-41bd-8252-b25c4c1ef048","Type":"ContainerStarted","Data":"c041382b85d6812dcd5bdf1b5ae9da982971376e60cf6eb6ee589406503a7734"} Nov 03 19:38:09.072440 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:09.072397 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-xp8mn" event={"ID":"89d72820-7b45-41bd-8252-b25c4c1ef048","Type":"ContainerStarted","Data":"f74e1f832cd75854c6ec771ae3e6437592e4bc544f9d58656b8611f0d3771557"} Nov 03 19:38:09.072879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:09.072564 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:09.101123 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:09.101073 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-xp8mn" podStartSLOduration=1.480040032 podStartE2EDuration="4.101061004s" totalImagesPullingTime="2.621020972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:09.100892096 +0000 UTC m=+396.805731439" watchObservedRunningTime="2026-11-03 19:38:09.101061004 +0000 UTC m=+396.805900343" Nov 03 19:38:20.079430 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:20.079353 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-xp8mn" Nov 03 19:38:21.988223 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:38:21.988186 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:38:22.937125 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:22.937094 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4qxjd"] Nov 03 19:38:22.941395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:22.941373 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:22.945096 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:22.945076 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 19:38:22.945210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:22.945132 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5cw4d\"" Nov 03 19:38:22.946378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:22.946363 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 19:38:22.948357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:22.948184 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 19:38:22.949805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:22.949784 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4qxjd"] Nov 03 19:38:23.099146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.099109 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-webhook-cert\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.099566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.099159 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-metrics-certs\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.099566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.099178 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddps6\" (UniqueName: \"kubernetes.io/projected/a0f7e007-32ac-46d4-957d-05fa7e811af5-kube-api-access-ddps6\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.200304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.200224 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-webhook-cert\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.200304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.200272 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-metrics-certs\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.200304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.200291 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddps6\" (UniqueName: \"kubernetes.io/projected/a0f7e007-32ac-46d4-957d-05fa7e811af5-kube-api-access-ddps6\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.202738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.202712 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-metrics-certs\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.202837 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.202783 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-webhook-cert\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.207915 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.207894 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddps6\" (UniqueName: \"kubernetes.io/projected/a0f7e007-32ac-46d4-957d-05fa7e811af5-kube-api-access-ddps6\") pod \"maas-controller-579cf7c6d-4qxjd\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.252184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.252161 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86c547b4b-8ftf8"] Nov 03 19:38:23.255483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.255466 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.255625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.255608 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:23.265652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.265629 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c547b4b-8ftf8"] Nov 03 19:38:23.402148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.402114 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-287q8\" (UniqueName: \"kubernetes.io/projected/98acad18-dee5-4a2b-9d7e-bf019690e3aa-kube-api-access-287q8\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.402305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.402161 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-metrics-certs\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.402305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.402185 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-webhook-cert\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.503237 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.503209 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-webhook-cert\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.503370 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.503275 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-287q8\" (UniqueName: \"kubernetes.io/projected/98acad18-dee5-4a2b-9d7e-bf019690e3aa-kube-api-access-287q8\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.503370 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.503305 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-metrics-certs\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.505483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.505460 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-webhook-cert\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.505611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.505503 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-metrics-certs\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.511402 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.511373 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-287q8\" (UniqueName: \"kubernetes.io/projected/98acad18-dee5-4a2b-9d7e-bf019690e3aa-kube-api-access-287q8\") pod \"maas-controller-86c547b4b-8ftf8\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.540783 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.540760 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4qxjd"] Nov 03 19:38:23.542999 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:38:23.542976 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f7e007_32ac_46d4_957d_05fa7e811af5.slice/crio-9d5d0587fb9e0162707af581bd06fddf14feac6a1498a5e7594f20f335ddf5fc WatchSource:0}: Error finding container 9d5d0587fb9e0162707af581bd06fddf14feac6a1498a5e7594f20f335ddf5fc: Status 404 returned error can't find the container with id 9d5d0587fb9e0162707af581bd06fddf14feac6a1498a5e7594f20f335ddf5fc Nov 03 19:38:23.583644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.583615 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:23.855991 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:23.855919 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c547b4b-8ftf8"] Nov 03 19:38:23.858917 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:38:23.858891 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98acad18_dee5_4a2b_9d7e_bf019690e3aa.slice/crio-f30aeeca09e68f9900247782d1b1fde4b0cd957021f8e68153e33e57ae0b4f72 WatchSource:0}: Error finding container f30aeeca09e68f9900247782d1b1fde4b0cd957021f8e68153e33e57ae0b4f72: Status 404 returned error can't find the container with id f30aeeca09e68f9900247782d1b1fde4b0cd957021f8e68153e33e57ae0b4f72 Nov 03 19:38:24.134616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:24.134514 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" event={"ID":"a0f7e007-32ac-46d4-957d-05fa7e811af5","Type":"ContainerStarted","Data":"9d5d0587fb9e0162707af581bd06fddf14feac6a1498a5e7594f20f335ddf5fc"} Nov 03 19:38:24.136086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:24.136055 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-8ftf8" event={"ID":"98acad18-dee5-4a2b-9d7e-bf019690e3aa","Type":"ContainerStarted","Data":"f30aeeca09e68f9900247782d1b1fde4b0cd957021f8e68153e33e57ae0b4f72"} Nov 03 19:38:27.152684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.152647 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-8ftf8" event={"ID":"98acad18-dee5-4a2b-9d7e-bf019690e3aa","Type":"ContainerStarted","Data":"be40491fce5400918243d8cd5b3e774de0a066fccae5abacc7998bcb3408cb6e"} Nov 03 19:38:27.153086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.152746 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:27.154109 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.154081 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" event={"ID":"a0f7e007-32ac-46d4-957d-05fa7e811af5","Type":"ContainerStarted","Data":"4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba"} Nov 03 19:38:27.154210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.154137 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:27.186512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.186460 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86c547b4b-8ftf8" podStartSLOduration=1.468937661 podStartE2EDuration="4.18644786s" totalImagesPullingTime="2.717510199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:27.184133918 +0000 UTC m=+414.888973259" watchObservedRunningTime="2026-11-03 19:38:27.18644786 +0000 UTC m=+414.891287200" Nov 03 19:38:27.208541 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.208496 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" podStartSLOduration=2.360562759 podStartE2EDuration="5.208484148s" totalImagesPullingTime="2.847921389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:27.207757533 +0000 UTC m=+414.912596873" watchObservedRunningTime="2026-11-03 19:38:27.208484148 +0000 UTC m=+414.913323488" Nov 03 19:38:27.976157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.976087 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz"] Nov 03 19:38:27.979687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.979663 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:27.987788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.987762 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 19:38:27.989049 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.989025 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 19:38:27.989223 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.989206 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-9p4xn\"" Nov 03 19:38:27.989268 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.989223 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 19:38:27.990889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:27.990862 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz"] Nov 03 19:38:28.041539 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.041502 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/900bc5dd-60bc-4597-9d8d-33c2551b5213-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.041715 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.041554 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900bc5dd-60bc-4597-9d8d-33c2551b5213-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.041715 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.041689 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/900bc5dd-60bc-4597-9d8d-33c2551b5213-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.041817 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.041790 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9sk\" (UniqueName: \"kubernetes.io/projected/900bc5dd-60bc-4597-9d8d-33c2551b5213-kube-api-access-tb9sk\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.142381 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.142345 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/900bc5dd-60bc-4597-9d8d-33c2551b5213-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.142574 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.142400 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9sk\" (UniqueName: \"kubernetes.io/projected/900bc5dd-60bc-4597-9d8d-33c2551b5213-kube-api-access-tb9sk\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.142574 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.142437 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/900bc5dd-60bc-4597-9d8d-33c2551b5213-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.142574 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.142477 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900bc5dd-60bc-4597-9d8d-33c2551b5213-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.143104 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.143078 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/900bc5dd-60bc-4597-9d8d-33c2551b5213-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.143277 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.143258 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/900bc5dd-60bc-4597-9d8d-33c2551b5213-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.144800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.144782 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/900bc5dd-60bc-4597-9d8d-33c2551b5213-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.150300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.150284 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9sk\" (UniqueName: \"kubernetes.io/projected/900bc5dd-60bc-4597-9d8d-33c2551b5213-kube-api-access-tb9sk\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6mwnz\" (UID: \"900bc5dd-60bc-4597-9d8d-33c2551b5213\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.300961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.300936 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:28.572871 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:28.572728 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz"] Nov 03 19:38:28.575400 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:38:28.575376 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900bc5dd_60bc_4597_9d8d_33c2551b5213.slice/crio-d740bde008c82958a28dcc91655246a10a01ca6f166bbaec11298260fec42395 WatchSource:0}: Error finding container d740bde008c82958a28dcc91655246a10a01ca6f166bbaec11298260fec42395: Status 404 returned error can't find the container with id d740bde008c82958a28dcc91655246a10a01ca6f166bbaec11298260fec42395 Nov 03 19:38:29.164888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:29.164854 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" event={"ID":"900bc5dd-60bc-4597-9d8d-33c2551b5213","Type":"ContainerStarted","Data":"d740bde008c82958a28dcc91655246a10a01ca6f166bbaec11298260fec42395"} Nov 03 19:38:38.162715 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.162683 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 19:38:38.163173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.163022 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:38.208448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.208404 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4qxjd"] Nov 03 19:38:38.213051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.212980 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" podUID="a0f7e007-32ac-46d4-957d-05fa7e811af5" containerName="manager" containerID="cri-o://4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba" gracePeriod=30 Nov 03 19:38:38.484504 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.478425 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:38.544593 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.544542 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-webhook-cert\") pod \"a0f7e007-32ac-46d4-957d-05fa7e811af5\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " Nov 03 19:38:38.544877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.544688 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-metrics-certs\") pod \"a0f7e007-32ac-46d4-957d-05fa7e811af5\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " Nov 03 19:38:38.544877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.544729 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f7e007-32ac-46d4-957d-05fa7e811af5-kube-api-access-ddps6\" (UniqueName: \"kubernetes.io/projected/a0f7e007-32ac-46d4-957d-05fa7e811af5-kube-api-access-ddps6\") pod \"a0f7e007-32ac-46d4-957d-05fa7e811af5\" (UID: \"a0f7e007-32ac-46d4-957d-05fa7e811af5\") " Nov 03 19:38:38.559281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.547206 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f7e007-32ac-46d4-957d-05fa7e811af5-kube-api-access-ddps6" pod "a0f7e007-32ac-46d4-957d-05fa7e811af5" (UID: "a0f7e007-32ac-46d4-957d-05fa7e811af5"). InnerVolumeSpecName "kube-api-access-ddps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:38:38.559281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.547254 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-webhook-cert" pod "a0f7e007-32ac-46d4-957d-05fa7e811af5" (UID: "a0f7e007-32ac-46d4-957d-05fa7e811af5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:38.559607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.559326 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-metrics-certs" pod "a0f7e007-32ac-46d4-957d-05fa7e811af5" (UID: "a0f7e007-32ac-46d4-957d-05fa7e811af5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:38:38.645775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.645734 4314 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-webhook-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:38.645775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.645770 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f7e007-32ac-46d4-957d-05fa7e811af5-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:38.646068 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:38.645784 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddps6\" (UniqueName: \"kubernetes.io/projected/a0f7e007-32ac-46d4-957d-05fa7e811af5-kube-api-access-ddps6\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:38:39.219544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.219500 4314 generic.go:356] "Generic (PLEG): container finished" podID="a0f7e007-32ac-46d4-957d-05fa7e811af5" containerID="4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba" exitCode=0 Nov 03 19:38:39.220058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.219597 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" Nov 03 19:38:39.220058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.219601 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" event={"ID":"a0f7e007-32ac-46d4-957d-05fa7e811af5","Type":"ContainerDied","Data":"4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba"} Nov 03 19:38:39.220058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.219765 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4qxjd" event={"ID":"a0f7e007-32ac-46d4-957d-05fa7e811af5","Type":"ContainerDied","Data":"9d5d0587fb9e0162707af581bd06fddf14feac6a1498a5e7594f20f335ddf5fc"} Nov 03 19:38:39.220058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.219790 4314 scope.go:122] "RemoveContainer" containerID="4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba" Nov 03 19:38:39.230760 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.230739 4314 scope.go:122] "RemoveContainer" containerID="4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba" Nov 03 19:38:39.231218 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:38:39.231189 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba\": container with ID starting with 4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba not found: ID does not exist" containerID="4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba" Nov 03 19:38:39.231303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.231222 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba"} err="failed to get container status \"4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba\": rpc error: code = NotFound desc = could not find container \"4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba\": container with ID starting with 4f51b67bf48b494bb3e8f86e70f8d8e50499e2d56866bae240aa89334d0192ba not found: ID does not exist" Nov 03 19:38:39.244246 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.244225 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4qxjd"] Nov 03 19:38:39.247422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:39.247393 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4qxjd"] Nov 03 19:38:40.225923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:40.225837 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" event={"ID":"900bc5dd-60bc-4597-9d8d-33c2551b5213","Type":"ContainerStarted","Data":"592c093bb33bce0a146b38d722e04e139785d172e6fb97bfc56a0327da3d13b5"} Nov 03 19:38:40.225923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:40.225902 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:40.242074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:40.242025 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" podStartSLOduration=2.024873185 podStartE2EDuration="13.242012726s" totalImagesPullingTime="11.217139541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:40.240197923 +0000 UTC m=+427.945037263" watchObservedRunningTime="2026-11-03 19:38:40.242012726 +0000 UTC m=+427.946852066" Nov 03 19:38:40.425698 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:40.425668 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f7e007-32ac-46d4-957d-05fa7e811af5" path="/var/lib/kubelet/pods/a0f7e007-32ac-46d4-957d-05fa7e811af5/volumes" Nov 03 19:38:51.236776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:38:51.236744 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" Nov 03 19:38:51.988552 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:38:51.988510 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:39:00.379309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.379276 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf"] Nov 03 19:39:00.379788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.379667 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a0f7e007-32ac-46d4-957d-05fa7e811af5" containerName="manager" Nov 03 19:39:00.379788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.379683 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f7e007-32ac-46d4-957d-05fa7e811af5" containerName="manager" Nov 03 19:39:00.379788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.379765 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f7e007-32ac-46d4-957d-05fa7e811af5" containerName="manager" Nov 03 19:39:00.387294 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.387270 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.391977 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.391638 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf"] Nov 03 19:39:00.395658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.395367 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rj6j4\"" Nov 03 19:39:00.395658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.395397 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 19:39:00.395658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.395503 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 19:39:00.395893 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.395866 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 19:39:00.395987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.395959 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 19:39:00.511397 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.511365 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztw2\" (UniqueName: \"kubernetes.io/projected/342d2ac9-80b9-4acf-ba26-cd369b090adf-kube-api-access-xztw2\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.511555 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.511417 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342d2ac9-80b9-4acf-ba26-cd369b090adf-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.511555 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.511440 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-maas-api-tls\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.511661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.511547 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-metrics-certs\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.612880 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.612854 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342d2ac9-80b9-4acf-ba26-cd369b090adf-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.613030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.612889 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-maas-api-tls\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.613030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.612916 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-metrics-certs\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.613030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.612977 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztw2\" (UniqueName: \"kubernetes.io/projected/342d2ac9-80b9-4acf-ba26-cd369b090adf-kube-api-access-xztw2\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.613626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.613577 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342d2ac9-80b9-4acf-ba26-cd369b090adf-openshift-service-ca\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.615416 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.615395 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-metrics-certs\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.615503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.615435 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-maas-api-tls\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.621198 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.621173 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztw2\" (UniqueName: \"kubernetes.io/projected/342d2ac9-80b9-4acf-ba26-cd369b090adf-kube-api-access-xztw2\") pod \"maas-api-6d669c7fcd-xsnwf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:00.703816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:00.703739 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:01.018210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:01.018186 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf"] Nov 03 19:39:01.020697 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:39:01.020667 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342d2ac9_80b9_4acf_ba26_cd369b090adf.slice/crio-19f95a48c4e4ea65bc760e59e774a2512a6c733e6dabd3a8521b0aa6ecb37e7f WatchSource:0}: Error finding container 19f95a48c4e4ea65bc760e59e774a2512a6c733e6dabd3a8521b0aa6ecb37e7f: Status 404 returned error can't find the container with id 19f95a48c4e4ea65bc760e59e774a2512a6c733e6dabd3a8521b0aa6ecb37e7f Nov 03 19:39:01.262871 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:01.262834 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:39:01.318033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:01.317940 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" event={"ID":"342d2ac9-80b9-4acf-ba26-cd369b090adf","Type":"ContainerStarted","Data":"19f95a48c4e4ea65bc760e59e774a2512a6c733e6dabd3a8521b0aa6ecb37e7f"} Nov 03 19:39:04.333377 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:04.333341 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" event={"ID":"342d2ac9-80b9-4acf-ba26-cd369b090adf","Type":"ContainerStarted","Data":"6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0"} Nov 03 19:39:04.333792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:04.333461 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:04.349520 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:04.349473 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" podStartSLOduration=1.886104698 podStartE2EDuration="4.34946203s" totalImagesPullingTime="2.463357332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:04.348865005 +0000 UTC m=+452.053704345" watchObservedRunningTime="2026-11-03 19:39:04.34946203 +0000 UTC m=+452.054301369" Nov 03 19:39:10.341997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:10.341970 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:11.222405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.222372 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl"] Nov 03 19:39:11.414189 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.414154 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl"] Nov 03 19:39:11.414550 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.414273 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.500616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.500559 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-openshift-service-ca\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.500773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.500703 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-maas-api-tls\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.500815 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.500769 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-metrics-certs\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.500850 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.500828 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvsc\" (UniqueName: \"kubernetes.io/projected/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-kube-api-access-5lvsc\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.601551 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.601492 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-maas-api-tls\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.601551 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.601557 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-metrics-certs\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.601776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.601652 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvsc\" (UniqueName: \"kubernetes.io/projected/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-kube-api-access-5lvsc\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.601776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.601689 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-openshift-service-ca\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.602422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.602395 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-openshift-service-ca\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.603791 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.603767 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-maas-api-tls\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.603932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.603912 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-metrics-certs\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.610725 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.610706 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvsc\" (UniqueName: \"kubernetes.io/projected/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-kube-api-access-5lvsc\") pod \"maas-api-6599bcbcd4-7z8xl\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:11.741249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:11.741217 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:12.082287 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:12.082094 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl"] Nov 03 19:39:12.085004 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:39:12.084974 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf7efc4_af20_4a85_90d6_e1ee635f9be1.slice/crio-02491e991d6d151a96d3b0f57ed8c243739c3cc905be3402e2b59808646ffd2a WatchSource:0}: Error finding container 02491e991d6d151a96d3b0f57ed8c243739c3cc905be3402e2b59808646ffd2a: Status 404 returned error can't find the container with id 02491e991d6d151a96d3b0f57ed8c243739c3cc905be3402e2b59808646ffd2a Nov 03 19:39:12.371333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:12.371255 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" event={"ID":"cbf7efc4-af20-4a85-90d6-e1ee635f9be1","Type":"ContainerStarted","Data":"02491e991d6d151a96d3b0f57ed8c243739c3cc905be3402e2b59808646ffd2a"} Nov 03 19:39:13.377451 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:13.377412 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" event={"ID":"cbf7efc4-af20-4a85-90d6-e1ee635f9be1","Type":"ContainerStarted","Data":"b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6"} Nov 03 19:39:13.377920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:13.377507 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:13.394484 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:13.394441 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" podStartSLOduration=1.7397966440000001 podStartE2EDuration="2.394427916s" totalImagesPullingTime="654.631272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:13.394070405 +0000 UTC m=+461.098909745" watchObservedRunningTime="2026-11-03 19:39:13.394427916 +0000 UTC m=+461.099267259" Nov 03 19:39:19.387063 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.387035 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 19:39:19.428613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.428556 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf"] Nov 03 19:39:19.428805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.428784 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" podUID="342d2ac9-80b9-4acf-ba26-cd369b090adf" containerName="maas-api" containerID="cri-o://6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0" gracePeriod=30 Nov 03 19:39:19.707038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.707013 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:19.870100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.870068 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-maas-api-tls\") pod \"342d2ac9-80b9-4acf-ba26-cd369b090adf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " Nov 03 19:39:19.870292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.870119 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342d2ac9-80b9-4acf-ba26-cd369b090adf-kube-api-access-xztw2\" (UniqueName: \"kubernetes.io/projected/342d2ac9-80b9-4acf-ba26-cd369b090adf-kube-api-access-xztw2\") pod \"342d2ac9-80b9-4acf-ba26-cd369b090adf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " Nov 03 19:39:19.870292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.870163 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342d2ac9-80b9-4acf-ba26-cd369b090adf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342d2ac9-80b9-4acf-ba26-cd369b090adf-openshift-service-ca\") pod \"342d2ac9-80b9-4acf-ba26-cd369b090adf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " Nov 03 19:39:19.870292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.870214 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-metrics-certs\") pod \"342d2ac9-80b9-4acf-ba26-cd369b090adf\" (UID: \"342d2ac9-80b9-4acf-ba26-cd369b090adf\") " Nov 03 19:39:19.870606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.870559 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342d2ac9-80b9-4acf-ba26-cd369b090adf-openshift-service-ca" pod "342d2ac9-80b9-4acf-ba26-cd369b090adf" (UID: "342d2ac9-80b9-4acf-ba26-cd369b090adf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:39:19.872379 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.872349 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-metrics-certs" pod "342d2ac9-80b9-4acf-ba26-cd369b090adf" (UID: "342d2ac9-80b9-4acf-ba26-cd369b090adf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:39:19.872463 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.872364 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-maas-api-tls" pod "342d2ac9-80b9-4acf-ba26-cd369b090adf" (UID: "342d2ac9-80b9-4acf-ba26-cd369b090adf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:39:19.872463 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.872410 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342d2ac9-80b9-4acf-ba26-cd369b090adf-kube-api-access-xztw2" pod "342d2ac9-80b9-4acf-ba26-cd369b090adf" (UID: "342d2ac9-80b9-4acf-ba26-cd369b090adf"). InnerVolumeSpecName "kube-api-access-xztw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:39:19.971251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.971182 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:19.971251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.971221 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/342d2ac9-80b9-4acf-ba26-cd369b090adf-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:19.971251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.971231 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztw2\" (UniqueName: \"kubernetes.io/projected/342d2ac9-80b9-4acf-ba26-cd369b090adf-kube-api-access-xztw2\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:19.971251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:19.971241 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/342d2ac9-80b9-4acf-ba26-cd369b090adf-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:39:20.409088 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.409051 4314 generic.go:356] "Generic (PLEG): container finished" podID="342d2ac9-80b9-4acf-ba26-cd369b090adf" containerID="6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0" exitCode=0 Nov 03 19:39:20.409517 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.409102 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" Nov 03 19:39:20.409517 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.409121 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" event={"ID":"342d2ac9-80b9-4acf-ba26-cd369b090adf","Type":"ContainerDied","Data":"6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0"} Nov 03 19:39:20.409517 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.409161 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf" event={"ID":"342d2ac9-80b9-4acf-ba26-cd369b090adf","Type":"ContainerDied","Data":"19f95a48c4e4ea65bc760e59e774a2512a6c733e6dabd3a8521b0aa6ecb37e7f"} Nov 03 19:39:20.409517 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.409185 4314 scope.go:122] "RemoveContainer" containerID="6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0" Nov 03 19:39:20.417771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.417737 4314 scope.go:122] "RemoveContainer" containerID="6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0" Nov 03 19:39:20.418029 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:39:20.418010 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0\": container with ID starting with 6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0 not found: ID does not exist" containerID="6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0" Nov 03 19:39:20.418076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.418037 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0"} err="failed to get container status \"6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0\": rpc error: code = NotFound desc = could not find container \"6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0\": container with ID starting with 6ffbefb5cad2484eff692035432aa6115511df8029d8ca8cc55515d5550bc3d0 not found: ID does not exist" Nov 03 19:39:20.429409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.429388 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf"] Nov 03 19:39:20.433617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:20.433595 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-xsnwf"] Nov 03 19:39:21.988624 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:39:21.988564 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:39:22.425489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:39:22.425459 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342d2ac9-80b9-4acf-ba26-cd369b090adf" path="/var/lib/kubelet/pods/342d2ac9-80b9-4acf-ba26-cd369b090adf/volumes" Nov 03 19:39:51.988889 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:39:51.988812 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:40:06.832897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:06.832861 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:40:08.595646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.595611 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97"] Nov 03 19:40:08.596110 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.595954 4314 cpu_manager.go:414] "containerMap: removing container" podUID="342d2ac9-80b9-4acf-ba26-cd369b090adf" containerName="maas-api" Nov 03 19:40:08.596110 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.595966 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342d2ac9-80b9-4acf-ba26-cd369b090adf" containerName="maas-api" Nov 03 19:40:08.596110 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.596053 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="342d2ac9-80b9-4acf-ba26-cd369b090adf" containerName="maas-api" Nov 03 19:40:08.599542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.599513 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.609405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.609379 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ctjfc\"" Nov 03 19:40:08.609526 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.609422 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 19:40:08.609526 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.609458 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 19:40:08.609526 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.609497 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 19:40:08.611527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.611504 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97"] Nov 03 19:40:08.662162 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.662134 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.662285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.662169 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.662285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.662193 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4973d4-3de9-4f1e-96cb-2def28c1396f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.662375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.662305 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.662375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.662359 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.662471 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.662455 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6wsh\" (UniqueName: \"kubernetes.io/projected/da4973d4-3de9-4f1e-96cb-2def28c1396f-kube-api-access-c6wsh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.763546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.763518 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.763678 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.763559 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6wsh\" (UniqueName: \"kubernetes.io/projected/da4973d4-3de9-4f1e-96cb-2def28c1396f-kube-api-access-c6wsh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.763723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.763694 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.763761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.763746 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.763793 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.763782 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4973d4-3de9-4f1e-96cb-2def28c1396f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.763852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.763836 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.763968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.763952 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.764034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.764015 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.764094 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.764079 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.766024 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.766000 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4973d4-3de9-4f1e-96cb-2def28c1396f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.766319 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.766297 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4973d4-3de9-4f1e-96cb-2def28c1396f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.781119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.781097 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6wsh\" (UniqueName: \"kubernetes.io/projected/da4973d4-3de9-4f1e-96cb-2def28c1396f-kube-api-access-c6wsh\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-sxc97\" (UID: \"da4973d4-3de9-4f1e-96cb-2def28c1396f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:08.914446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:08.914367 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:09.191263 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:09.191197 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97"] Nov 03 19:40:09.193953 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:09.193925 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4973d4_3de9_4f1e_96cb_2def28c1396f.slice/crio-9644b2adb1cf82d346f0ac20a6c863f397248b9a776b98296fdd854f0d2982cf WatchSource:0}: Error finding container 9644b2adb1cf82d346f0ac20a6c863f397248b9a776b98296fdd854f0d2982cf: Status 404 returned error can't find the container with id 9644b2adb1cf82d346f0ac20a6c863f397248b9a776b98296fdd854f0d2982cf Nov 03 19:40:09.204485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:09.204459 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:40:09.612356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:09.612283 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" event={"ID":"da4973d4-3de9-4f1e-96cb-2def28c1396f","Type":"ContainerStarted","Data":"9644b2adb1cf82d346f0ac20a6c863f397248b9a776b98296fdd854f0d2982cf"} Nov 03 19:40:10.619551 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:10.619512 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" event={"ID":"da4973d4-3de9-4f1e-96cb-2def28c1396f","Type":"ContainerStarted","Data":"0da2e61a2fdb9c33df2b9c5f4da4b9228e4e62ddbe9b7a0f398ac99ac52c228d"} Nov 03 19:40:10.637059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:10.637015 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" podStartSLOduration=1.438999659 podStartE2EDuration="2.637002181s" totalImagesPullingTime="1.198002522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:10.635284342 +0000 UTC m=+518.340123693" watchObservedRunningTime="2026-11-03 19:40:10.637002181 +0000 UTC m=+518.341841522" Nov 03 19:40:10.915000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:10.914966 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:11.734100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:11.734064 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:40:13.005521 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.005490 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95"] Nov 03 19:40:13.009526 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.009496 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.019031 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.019007 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 19:40:13.020428 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.020407 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95"] Nov 03 19:40:13.105285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.105244 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.105285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.105281 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.105523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.105373 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.105523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.105497 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.105637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.105532 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.105637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.105567 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddww9\" (UniqueName: \"kubernetes.io/projected/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-kube-api-access-ddww9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206241 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206207 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206241 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206239 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206258 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206386 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206432 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206473 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddww9\" (UniqueName: \"kubernetes.io/projected/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-kube-api-access-ddww9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206707 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206778 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206745 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.206872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.206849 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.208698 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.208672 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.209507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.209488 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.213981 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.213958 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddww9\" (UniqueName: \"kubernetes.io/projected/3e2993e6-3c87-42aa-9279-fc3d11d9e1c2-kube-api-access-ddww9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95\" (UID: \"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.326182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.326114 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:13.595921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.595866 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95"] Nov 03 19:40:13.598393 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:13.598366 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2993e6_3c87_42aa_9279_fc3d11d9e1c2.slice/crio-ded1de341b4d91c937153cb3f477b4706ca130c3ab6bcda7f39e686dd89b8541 WatchSource:0}: Error finding container ded1de341b4d91c937153cb3f477b4706ca130c3ab6bcda7f39e686dd89b8541: Status 404 returned error can't find the container with id ded1de341b4d91c937153cb3f477b4706ca130c3ab6bcda7f39e686dd89b8541 Nov 03 19:40:13.632411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:13.632379 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" event={"ID":"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2","Type":"ContainerStarted","Data":"ded1de341b4d91c937153cb3f477b4706ca130c3ab6bcda7f39e686dd89b8541"} Nov 03 19:40:14.311225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:14.311190 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:40:14.638077 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:14.637989 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" event={"ID":"3e2993e6-3c87-42aa-9279-fc3d11d9e1c2","Type":"ContainerStarted","Data":"24f4993ed8c706d857a77b1824cc08b9a788de3eeccb2bd1f6b3e9cb86c92376"} Nov 03 19:40:14.661558 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:14.661513 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" podStartSLOduration=2.356330717 podStartE2EDuration="2.661500428s" totalImagesPullingTime="305.169711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:14.659530799 +0000 UTC m=+522.364370141" watchObservedRunningTime="2026-11-03 19:40:14.661500428 +0000 UTC m=+522.366339768" Nov 03 19:40:15.326595 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:15.326564 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:15.921035 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:15.921001 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m"] Nov 03 19:40:15.925595 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:15.925553 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:15.934135 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:15.934115 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 19:40:15.936279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:15.936255 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m"] Nov 03 19:40:16.029102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.029066 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab1ff97-e41f-4a1b-900e-768abaf58008-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.029266 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.029117 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2bs\" (UniqueName: \"kubernetes.io/projected/6ab1ff97-e41f-4a1b-900e-768abaf58008-kube-api-access-4j2bs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.029266 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.029184 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.029266 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.029221 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.029266 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.029239 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.029417 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.029340 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.130445 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130407 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.130445 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130447 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.130740 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130467 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.130740 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130575 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.130740 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130695 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab1ff97-e41f-4a1b-900e-768abaf58008-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.130906 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130757 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2bs\" (UniqueName: \"kubernetes.io/projected/6ab1ff97-e41f-4a1b-900e-768abaf58008-kube-api-access-4j2bs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.130988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130965 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.131047 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.130993 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.131082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.131046 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.132964 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.132943 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab1ff97-e41f-4a1b-900e-768abaf58008-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.133212 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.133186 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab1ff97-e41f-4a1b-900e-768abaf58008-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.138220 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.138198 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2bs\" (UniqueName: \"kubernetes.io/projected/6ab1ff97-e41f-4a1b-900e-768abaf58008-kube-api-access-4j2bs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m\" (UID: \"6ab1ff97-e41f-4a1b-900e-768abaf58008\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.241107 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.241078 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:16.510980 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.510829 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m"] Nov 03 19:40:16.513848 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:16.513815 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab1ff97_e41f_4a1b_900e_768abaf58008.slice/crio-376feb38c19d50a626fb56e38e0f4df208b0300dfa761ea4a1fd427dd1be0b76 WatchSource:0}: Error finding container 376feb38c19d50a626fb56e38e0f4df208b0300dfa761ea4a1fd427dd1be0b76: Status 404 returned error can't find the container with id 376feb38c19d50a626fb56e38e0f4df208b0300dfa761ea4a1fd427dd1be0b76 Nov 03 19:40:16.653075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:16.653035 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" event={"ID":"6ab1ff97-e41f-4a1b-900e-768abaf58008","Type":"ContainerStarted","Data":"376feb38c19d50a626fb56e38e0f4df208b0300dfa761ea4a1fd427dd1be0b76"} Nov 03 19:40:17.659196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:17.659159 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" event={"ID":"6ab1ff97-e41f-4a1b-900e-768abaf58008","Type":"ContainerStarted","Data":"7223d752dbebaf981bf38a0767f34a6031824b0c3eb77306cbb6e0976c21c4a7"} Nov 03 19:40:17.677307 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:17.677258 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" podStartSLOduration=2.337859189 podStartE2EDuration="2.67724471s" totalImagesPullingTime="339.385521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:17.675256705 +0000 UTC m=+525.380096046" watchObservedRunningTime="2026-11-03 19:40:17.67724471 +0000 UTC m=+525.382084051" Nov 03 19:40:18.241970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:18.241932 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:18.448028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:18.447996 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:40:18.915081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:18.915041 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:18.927524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:18.927501 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:19.687322 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:19.687288 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" Nov 03 19:40:20.063786 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.063744 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl"] Nov 03 19:40:20.067906 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.067884 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.076720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.076689 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl"] Nov 03 19:40:20.078411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.078381 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 19:40:20.169383 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.169344 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hcrv\" (UniqueName: \"kubernetes.io/projected/0067d07b-adb9-4976-a4fb-6bdcec9091cf-kube-api-access-5hcrv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.169566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.169394 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.169566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.169422 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.169566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.169473 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.169706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.169667 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.169741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.169725 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0067d07b-adb9-4976-a4fb-6bdcec9091cf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.270823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.270784 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0067d07b-adb9-4976-a4fb-6bdcec9091cf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.270866 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcrv\" (UniqueName: \"kubernetes.io/projected/0067d07b-adb9-4976-a4fb-6bdcec9091cf-kube-api-access-5hcrv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.270905 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.270930 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271243 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.271218 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271423 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.271404 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271519 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.271434 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271519 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.271486 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.271822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.271802 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.273787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.273758 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0067d07b-adb9-4976-a4fb-6bdcec9091cf-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.274003 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.273987 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0067d07b-adb9-4976-a4fb-6bdcec9091cf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.278781 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.278761 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcrv\" (UniqueName: \"kubernetes.io/projected/0067d07b-adb9-4976-a4fb-6bdcec9091cf-kube-api-access-5hcrv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl\" (UID: \"0067d07b-adb9-4976-a4fb-6bdcec9091cf\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.386322 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.386238 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:20.664818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.664789 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl"] Nov 03 19:40:20.667765 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:20.667734 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0067d07b_adb9_4976_a4fb_6bdcec9091cf.slice/crio-fd314cc10c20031d1be2217f88fdcd12968745e198f64d261dc51ae6d6e2a5c6 WatchSource:0}: Error finding container fd314cc10c20031d1be2217f88fdcd12968745e198f64d261dc51ae6d6e2a5c6: Status 404 returned error can't find the container with id fd314cc10c20031d1be2217f88fdcd12968745e198f64d261dc51ae6d6e2a5c6 Nov 03 19:40:20.679270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:20.679243 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" event={"ID":"0067d07b-adb9-4976-a4fb-6bdcec9091cf","Type":"ContainerStarted","Data":"fd314cc10c20031d1be2217f88fdcd12968745e198f64d261dc51ae6d6e2a5c6"} Nov 03 19:40:21.438385 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:21.438303 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:40:21.686783 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:21.686743 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" event={"ID":"0067d07b-adb9-4976-a4fb-6bdcec9091cf","Type":"ContainerStarted","Data":"9ce01721645e889be256e7b64697c5eac7bfab70c500898de2472a5a295b65e3"} Nov 03 19:40:21.721487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:21.721392 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" podStartSLOduration=1.36113022 podStartE2EDuration="1.721376848s" totalImagesPullingTime="360.246628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:21.718568615 +0000 UTC m=+529.423407958" watchObservedRunningTime="2026-11-03 19:40:21.721376848 +0000 UTC m=+529.426216211" Nov 03 19:40:21.990460 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:40:21.990375 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:40:22.387185 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:22.387152 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:23.327223 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:23.327192 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:23.339643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:23.339621 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:23.706799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:23.706729 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" Nov 03 19:40:26.241997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:26.241959 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:26.254546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:26.254522 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:26.721201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:26.721131 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m" Nov 03 19:40:30.386567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:30.386524 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:30.399149 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:30.399124 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:30.738784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:30.738709 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" Nov 03 19:40:38.743643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.743608 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc"] Nov 03 19:40:38.748702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.748681 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.761228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.761198 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 19:40:38.763082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.763055 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc"] Nov 03 19:40:38.830197 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830168 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4k4\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-kube-api-access-hn4k4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830203 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830227 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830410 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830381 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830408 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830422 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830560 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830639 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.830731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.830667 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.843756 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.843737 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7lc4h\"" Nov 03 19:40:38.931515 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931491 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931526 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931543 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931569 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4k4\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-kube-api-access-hn4k4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931628 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931661 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931723 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931760 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.931853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.931782 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.932153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.932131 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.932235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.932187 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.932314 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.932276 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.932364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.932327 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.932364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.932281 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.934123 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.934106 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.934187 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.934155 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.939689 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.939669 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:38.939893 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:38.939870 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4k4\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-kube-api-access-hn4k4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-76cxc\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:39.061565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:39.061502 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:39.344382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:39.344178 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc"] Nov 03 19:40:39.346993 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:39.346964 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e969e0_d5d8_42d3_a243_fdba3ec8534f.slice/crio-52ef1d6df3dca96232e9bed0e5e1dd0770a7d3aaed984d5fb146d8e780b88264 WatchSource:0}: Error finding container 52ef1d6df3dca96232e9bed0e5e1dd0770a7d3aaed984d5fb146d8e780b88264: Status 404 returned error can't find the container with id 52ef1d6df3dca96232e9bed0e5e1dd0770a7d3aaed984d5fb146d8e780b88264 Nov 03 19:40:39.771536 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:39.771498 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" event={"ID":"b7e969e0-d5d8-42d3-a243-fdba3ec8534f","Type":"ContainerStarted","Data":"52ef1d6df3dca96232e9bed0e5e1dd0770a7d3aaed984d5fb146d8e780b88264"} Nov 03 19:40:40.651681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.651648 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-76c4k"] Nov 03 19:40:40.656268 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.656248 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:40.661143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.661124 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 19:40:40.662752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.662732 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bmn5w\"" Nov 03 19:40:40.664513 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.664489 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-76c4k"] Nov 03 19:40:40.701238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.701207 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-j9kp5"] Nov 03 19:40:40.711624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.705134 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:40.714207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.714180 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-j9kp5"] Nov 03 19:40:40.749371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.749346 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8kv\" (UniqueName: \"kubernetes.io/projected/501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a-kube-api-access-rm8kv\") pod \"payload-processing-55fbbbf9dc-j9kp5\" (UID: \"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:40.749624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.749379 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-j9kp5\" (UID: \"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:40.749624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.749449 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55qss\" (UniqueName: \"kubernetes.io/projected/1205a003-d0a7-4aa0-8cdd-b38e0afd4e63-kube-api-access-55qss\") pod \"payload-pre-processing-7b98bb678c-76c4k\" (UID: \"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:40.749624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.749496 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205a003-d0a7-4aa0-8cdd-b38e0afd4e63-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-76c4k\" (UID: \"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:40.850643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.850613 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8kv\" (UniqueName: \"kubernetes.io/projected/501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a-kube-api-access-rm8kv\") pod \"payload-processing-55fbbbf9dc-j9kp5\" (UID: \"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:40.850643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.850649 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-j9kp5\" (UID: \"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:40.851211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.850701 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55qss\" (UniqueName: \"kubernetes.io/projected/1205a003-d0a7-4aa0-8cdd-b38e0afd4e63-kube-api-access-55qss\") pod \"payload-pre-processing-7b98bb678c-76c4k\" (UID: \"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:40.851211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.850732 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205a003-d0a7-4aa0-8cdd-b38e0afd4e63-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-76c4k\" (UID: \"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:40.851436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.851416 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205a003-d0a7-4aa0-8cdd-b38e0afd4e63-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-76c4k\" (UID: \"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:40.851615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.851571 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-j9kp5\" (UID: \"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:40.858895 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.858871 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8kv\" (UniqueName: \"kubernetes.io/projected/501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a-kube-api-access-rm8kv\") pod \"payload-processing-55fbbbf9dc-j9kp5\" (UID: \"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:40.859011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.858951 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55qss\" (UniqueName: \"kubernetes.io/projected/1205a003-d0a7-4aa0-8cdd-b38e0afd4e63-kube-api-access-55qss\") pod \"payload-pre-processing-7b98bb678c-76c4k\" (UID: \"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:40.975392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:40.975309 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:41.031714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:41.029757 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:41.247813 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:41.247788 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-76c4k"] Nov 03 19:40:41.250162 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:41.250133 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1205a003_d0a7_4aa0_8cdd_b38e0afd4e63.slice/crio-418810df9857a41c504ecd92179172559fd969d0367539f9f0ab6893b8b29ea0 WatchSource:0}: Error finding container 418810df9857a41c504ecd92179172559fd969d0367539f9f0ab6893b8b29ea0: Status 404 returned error can't find the container with id 418810df9857a41c504ecd92179172559fd969d0367539f9f0ab6893b8b29ea0 Nov 03 19:40:41.306725 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:41.306702 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-j9kp5"] Nov 03 19:40:41.308548 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:41.308522 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501a6ce3_bb3e_42ac_84cf_7bb98fb19d5a.slice/crio-106379cc9c12bd4c2ae236ec17f18d0c0fbfc0534b4c8d76dd73f58bcc6e24e7 WatchSource:0}: Error finding container 106379cc9c12bd4c2ae236ec17f18d0c0fbfc0534b4c8d76dd73f58bcc6e24e7: Status 404 returned error can't find the container with id 106379cc9c12bd4c2ae236ec17f18d0c0fbfc0534b4c8d76dd73f58bcc6e24e7 Nov 03 19:40:41.782863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:41.782805 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" event={"ID":"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a","Type":"ContainerStarted","Data":"106379cc9c12bd4c2ae236ec17f18d0c0fbfc0534b4c8d76dd73f58bcc6e24e7"} Nov 03 19:40:41.785759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:41.785724 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" event={"ID":"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63","Type":"ContainerStarted","Data":"418810df9857a41c504ecd92179172559fd969d0367539f9f0ab6893b8b29ea0"} Nov 03 19:40:44.804764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:44.804720 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" event={"ID":"501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a","Type":"ContainerStarted","Data":"888759ae3abf068343e2df47624adb37d20e3e133b197f56776eea801642b72b"} Nov 03 19:40:44.805238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:44.804820 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:44.806599 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:44.806545 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" event={"ID":"1205a003-d0a7-4aa0-8cdd-b38e0afd4e63","Type":"ContainerStarted","Data":"b38ca224110862f076d60f04c62fbc131f6d951c39782c4acc208f7dffaf8080"} Nov 03 19:40:44.806709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:44.806628 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:44.821230 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:44.821183 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" podStartSLOduration=2.188640008 podStartE2EDuration="4.821170276s" totalImagesPullingTime="2.632530268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:44.82022945 +0000 UTC m=+552.525068805" watchObservedRunningTime="2026-11-03 19:40:44.821170276 +0000 UTC m=+552.526009616" Nov 03 19:40:44.834665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:44.834620 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" podStartSLOduration=2.001512173 podStartE2EDuration="4.834606378s" totalImagesPullingTime="2.833094205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:44.834142735 +0000 UTC m=+552.538982076" watchObservedRunningTime="2026-11-03 19:40:44.834606378 +0000 UTC m=+552.539445720" Nov 03 19:40:45.253566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:45.253528 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:45.253710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:45.253626 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:45.253710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:45.253677 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:45.816614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:45.816559 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" event={"ID":"b7e969e0-d5d8-42d3-a243-fdba3ec8534f","Type":"ContainerStarted","Data":"1c2844bdafa19e6f2c0dbcdcb434d9ed0d49f55cb49586281f346cf8209ee7c6"} Nov 03 19:40:45.838143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:45.838097 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podStartSLOduration=1.934228413 podStartE2EDuration="7.83808409s" totalImagesPullingTime="5.903855677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:45.836198047 +0000 UTC m=+553.541037387" watchObservedRunningTime="2026-11-03 19:40:45.83808409 +0000 UTC m=+553.542923413" Nov 03 19:40:46.077660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:46.076968 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:46.081761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:46.081722 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 19:40:46.081927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:46.081812 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 19:40:47.067939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:47.067902 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 19:40:47.068500 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:47.068470 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 19:40:48.071000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:48.070960 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 19:40:48.071569 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:48.071535 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 19:40:49.075604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:49.069947 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 19:40:49.075604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:49.070010 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 19:40:50.073443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:50.072982 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 19:40:50.073443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:50.073050 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 19:40:51.081609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:51.078406 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 19:40:51.081609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:51.078464 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 19:40:51.988713 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:40:51.988672 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:40:52.062119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:52.062083 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 19:40:52.062284 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:52.062162 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 19:40:53.545081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.545037 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:53.547486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.545743 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:53.547486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.546493 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 19:40:53.760339 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.760301 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx"] Nov 03 19:40:53.763696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.763680 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.773442 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.773418 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx"] Nov 03 19:40:53.914487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914403 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914443 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914470 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwp5w\" (UniqueName: \"kubernetes.io/projected/361708af-2ed1-40e2-8da6-a3e28562ce7b-kube-api-access-cwp5w\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914557 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914603 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914701 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914815 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914728 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914815 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914758 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361708af-2ed1-40e2-8da6-a3e28562ce7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:53.914815 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:53.914798 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015458 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015486 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015511 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361708af-2ed1-40e2-8da6-a3e28562ce7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015643 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015780 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015833 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015911 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015833 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015911 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015879 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwp5w\" (UniqueName: \"kubernetes.io/projected/361708af-2ed1-40e2-8da6-a3e28562ce7b-kube-api-access-cwp5w\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.015911 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015890 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.016084 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015911 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.016084 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015934 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.016084 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.015988 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.016248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.016078 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.016288 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.016268 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361708af-2ed1-40e2-8da6-a3e28562ce7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.018242 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.018211 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.018359 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.018344 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.024572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.024551 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361708af-2ed1-40e2-8da6-a3e28562ce7b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.024699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.024679 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwp5w\" (UniqueName: \"kubernetes.io/projected/361708af-2ed1-40e2-8da6-a3e28562ce7b-kube-api-access-cwp5w\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lx2rx\" (UID: \"361708af-2ed1-40e2-8da6-a3e28562ce7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.078978 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.078949 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:54.359637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.359610 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx"] Nov 03 19:40:54.362314 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:40:54.362284 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361708af_2ed1_40e2_8da6_a3e28562ce7b.slice/crio-16a48ef1bec2072dc020445541512797302bc737ab51b0f3074242ddd4930bee WatchSource:0}: Error finding container 16a48ef1bec2072dc020445541512797302bc737ab51b0f3074242ddd4930bee: Status 404 returned error can't find the container with id 16a48ef1bec2072dc020445541512797302bc737ab51b0f3074242ddd4930bee Nov 03 19:40:54.365437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.365404 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:54.365513 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.365470 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:54.365513 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.365498 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:40:54.874762 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.874725 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" event={"ID":"361708af-2ed1-40e2-8da6-a3e28562ce7b","Type":"ContainerStarted","Data":"3e8d010ad8204653df7eb06ec54e1ce43dc285fb61e245803f3271788b4c1ca0"} Nov 03 19:40:54.875415 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.875389 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" event={"ID":"361708af-2ed1-40e2-8da6-a3e28562ce7b","Type":"ContainerStarted","Data":"16a48ef1bec2072dc020445541512797302bc737ab51b0f3074242ddd4930bee"} Nov 03 19:40:54.896521 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:54.896455 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podStartSLOduration=1.8964415749999999 podStartE2EDuration="1.896441575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:54.894394515 +0000 UTC m=+562.599233882" watchObservedRunningTime="2026-11-03 19:40:54.896441575 +0000 UTC m=+562.601280912" Nov 03 19:40:55.082005 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:55.081956 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:40:55.083628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:55.083566 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 19:40:55.083768 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:55.083662 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 19:40:55.819603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:55.819562 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-76c4k" Nov 03 19:40:55.819785 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:55.819656 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-j9kp5" Nov 03 19:40:56.082047 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:56.081956 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 19:40:56.082047 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:56.082010 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 19:40:57.082039 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:57.081994 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 19:40:57.082513 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:57.082061 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 19:40:58.082029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:58.081989 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 19:40:58.082210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:58.082048 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 19:40:59.083049 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:59.083009 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 19:40:59.083481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:40:59.083079 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 19:41:00.082005 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:00.081966 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 19:41:00.082205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:00.082035 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 19:41:01.079963 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:01.079921 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 19:41:01.080341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:01.080005 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 19:41:02.571881 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:02.571851 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:41:02.572343 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:02.572161 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:41:02.573078 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:02.573055 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" Nov 03 19:41:08.784553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.784522 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj"] Nov 03 19:41:08.788729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.788702 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.800620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.800596 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj"] Nov 03 19:41:08.882649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882616 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202ad091-28ed-41c2-aeae-d60761a4d034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.882649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882651 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.882851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882669 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202ad091-28ed-41c2-aeae-d60761a4d034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.882851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882781 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.882851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882838 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.882949 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882854 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.882949 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882873 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.882949 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882914 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.883045 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.882974 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nm4\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-kube-api-access-p7nm4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983496 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983465 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202ad091-28ed-41c2-aeae-d60761a4d034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983496 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983495 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983511 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202ad091-28ed-41c2-aeae-d60761a4d034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983531 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983575 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983631 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983677 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.983759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983731 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.984054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.983764 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nm4\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-kube-api-access-p7nm4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.984054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.984028 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.984225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.984083 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.984225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.984197 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.984328 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.984275 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.984386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.984330 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202ad091-28ed-41c2-aeae-d60761a4d034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.986011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.985990 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.986247 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.986225 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202ad091-28ed-41c2-aeae-d60761a4d034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.991912 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.991893 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:08.991997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:08.991909 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nm4\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-kube-api-access-p7nm4\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-57ktj\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:09.103485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.103413 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:09.388946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.388922 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj"] Nov 03 19:41:09.391641 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:09.391609 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202ad091_28ed_41c2_aeae_d60761a4d034.slice/crio-19281904efae9196619d4cb6bb73d6b568a0e8cd8c13646c2ee0698b0f2ae302 WatchSource:0}: Error finding container 19281904efae9196619d4cb6bb73d6b568a0e8cd8c13646c2ee0698b0f2ae302: Status 404 returned error can't find the container with id 19281904efae9196619d4cb6bb73d6b568a0e8cd8c13646c2ee0698b0f2ae302 Nov 03 19:41:09.394229 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.394190 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:09.394307 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.394290 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:09.394361 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.394343 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:09.951931 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.951884 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" event={"ID":"202ad091-28ed-41c2-aeae-d60761a4d034","Type":"ContainerStarted","Data":"8e05e8c3ddecd72dadafacf4ac13b1387aaf034591a5d95ed53c30c40a053df9"} Nov 03 19:41:09.952386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.951945 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" event={"ID":"202ad091-28ed-41c2-aeae-d60761a4d034","Type":"ContainerStarted","Data":"19281904efae9196619d4cb6bb73d6b568a0e8cd8c13646c2ee0698b0f2ae302"} Nov 03 19:41:09.979444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:09.979377 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podStartSLOduration=1.979358757 podStartE2EDuration="1.979358757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:09.977651797 +0000 UTC m=+577.682491137" watchObservedRunningTime="2026-11-03 19:41:09.979358757 +0000 UTC m=+577.684198098" Nov 03 19:41:10.104565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:10.104530 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:10.106245 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:10.106215 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 19:41:10.106360 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:10.106262 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 19:41:11.104201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:11.104165 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 19:41:11.104637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:11.104233 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 19:41:12.104611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:12.104560 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 19:41:12.105026 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:12.104636 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 19:41:13.104175 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:13.104139 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 19:41:13.104347 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:13.104195 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 19:41:14.104135 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:14.104097 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 19:41:14.104543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:14.104155 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 19:41:15.104189 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:15.104111 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 19:41:15.104614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:15.104195 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 19:41:16.104771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:16.104726 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 19:41:16.105190 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:16.104787 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 19:41:17.590073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:17.590041 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:17.590917 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:17.590891 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:17.591362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:17.591339 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 19:41:21.989270 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:41:21.989228 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:41:32.150848 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.150813 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn"] Nov 03 19:41:32.155228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.155203 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.167075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.167035 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn"] Nov 03 19:41:32.167764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.167743 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-39043d-openshift-default-dockercfg-zwlqs\"" Nov 03 19:41:32.230994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.230960 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k"] Nov 03 19:41:32.234600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.234559 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.245979 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.245953 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k"] Nov 03 19:41:32.267946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.267914 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a35658-openshift-default-dockercfg-m75kt\"" Nov 03 19:41:32.307995 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.307965 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425h9\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-kube-api-access-425h9\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308017 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308145 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308286 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308182 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzcd\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-kube-api-access-mpzcd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308286 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308225 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308286 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308282 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308312 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308363 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308392 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308423 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308449 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308480 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308560 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308603 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308652 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308688 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.308812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308715 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.308812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.308749 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409734 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409791 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409849 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409875 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409908 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409935 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409959 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.409983 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410016 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425h9\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-kube-api-access-425h9\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410055 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410104 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410132 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzcd\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-kube-api-access-mpzcd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410175 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410193 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410217 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410243 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410283 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410310 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410340 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.410744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410512 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.411920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410824 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.411920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.410985 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.411920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.411213 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.411920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.411750 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.412123 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.411978 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.412316 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.412282 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.412425 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.412395 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.413260 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.413230 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.414023 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.413991 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.414634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.414573 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.416434 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.416411 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.417495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.417474 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.431549 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.431521 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.431716 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.431690 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzcd\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-kube-api-access-mpzcd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.433639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.433501 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.435192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.435165 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425h9\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-kube-api-access-425h9\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.490054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.490016 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-39043d-openshift-default-dockercfg-zwlqs\"" Nov 03 19:41:32.497885 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.497841 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:41:32.499488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.499404 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:41:32.499814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.499795 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:32.556522 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.556447 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a35658-openshift-default-dockercfg-m75kt\"" Nov 03 19:41:32.563971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.563926 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:32.678310 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.678235 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7"] Nov 03 19:41:32.682771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.682751 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.699245 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.697414 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7"] Nov 03 19:41:32.705615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.705577 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-102379-openshift-default-dockercfg-2s5bm\"" Nov 03 19:41:32.791334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.791310 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn"] Nov 03 19:41:32.793321 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:32.793294 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d965a1_421d_4ea9_8d6a_ac95bb9e8ea1.slice/crio-07d3584bbb46c42ddd33b619f1c17984adaea7a13b85376fc8e383a844a8bc03 WatchSource:0}: Error finding container 07d3584bbb46c42ddd33b619f1c17984adaea7a13b85376fc8e383a844a8bc03: Status 404 returned error can't find the container with id 07d3584bbb46c42ddd33b619f1c17984adaea7a13b85376fc8e383a844a8bc03 Nov 03 19:41:32.795636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.795605 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:32.795709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.795672 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:32.795709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.795698 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:32.816210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816185 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816302 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816217 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816355 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816339 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816375 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816452 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816394 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7mb\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-kube-api-access-lm7mb\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816452 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816417 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816545 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816529 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816576 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.816706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.816687 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.868489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.868460 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k"] Nov 03 19:41:32.871918 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:32.871884 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447a4a3a_9e30_44ae_b196_f9c2a7b77fc9.slice/crio-765b34cac7cb06440202618dc0c29487f829dcfd88bc5ee2f1fc616a07843921 WatchSource:0}: Error finding container 765b34cac7cb06440202618dc0c29487f829dcfd88bc5ee2f1fc616a07843921: Status 404 returned error can't find the container with id 765b34cac7cb06440202618dc0c29487f829dcfd88bc5ee2f1fc616a07843921 Nov 03 19:41:32.874835 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.874790 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:32.874930 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.874884 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:32.874994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.874927 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:32.920675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.917705 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.920925 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.920899 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.921100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.921079 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.921215 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.921201 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.921391 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.921377 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.921499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.921486 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.921605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.921574 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7mb\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-kube-api-access-lm7mb\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.921736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.921722 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.922378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.921848 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.923178 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.923155 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.926569 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.923594 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.926730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.923809 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.926830 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.924064 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.927029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.924773 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.927125 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.925247 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.930133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.930058 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.939625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.938768 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:32.943812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:32.941985 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7mb\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-kube-api-access-lm7mb\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:33.007756 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.007721 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:33.096103 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.095121 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" event={"ID":"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9","Type":"ContainerStarted","Data":"bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51"} Nov 03 19:41:33.096103 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.095190 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" event={"ID":"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9","Type":"ContainerStarted","Data":"765b34cac7cb06440202618dc0c29487f829dcfd88bc5ee2f1fc616a07843921"} Nov 03 19:41:33.109075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.109042 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" event={"ID":"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1","Type":"ContainerStarted","Data":"5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d"} Nov 03 19:41:33.109214 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.109102 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" event={"ID":"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1","Type":"ContainerStarted","Data":"07d3584bbb46c42ddd33b619f1c17984adaea7a13b85376fc8e383a844a8bc03"} Nov 03 19:41:33.127621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.124277 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podStartSLOduration=1.124257803 podStartE2EDuration="1.124257803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:33.123911341 +0000 UTC m=+600.828750688" watchObservedRunningTime="2026-11-03 19:41:33.124257803 +0000 UTC m=+600.829097147" Nov 03 19:41:33.147304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.147244 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podStartSLOduration=1.147224983 podStartE2EDuration="1.147224983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:33.144666435 +0000 UTC m=+600.849505776" watchObservedRunningTime="2026-11-03 19:41:33.147224983 +0000 UTC m=+600.852064324" Nov 03 19:41:33.322303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.322271 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7"] Nov 03 19:41:33.323528 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:33.323501 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9cbe6d_f94f_4d28_95df_a55517f3c4ba.slice/crio-1ba2b5c1eb3b1fb9a9fb281d81158f0d88f99a887ab07f05bfcfdcc3a747c1e2 WatchSource:0}: Error finding container 1ba2b5c1eb3b1fb9a9fb281d81158f0d88f99a887ab07f05bfcfdcc3a747c1e2: Status 404 returned error can't find the container with id 1ba2b5c1eb3b1fb9a9fb281d81158f0d88f99a887ab07f05bfcfdcc3a747c1e2 Nov 03 19:41:33.326064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.326026 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:33.326160 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.326103 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:33.326160 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.326146 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:33.500954 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.500925 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:33.506367 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.506335 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:33.577658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.568870 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:33.586333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:33.584912 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:34.115891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:34.115853 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" event={"ID":"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba","Type":"ContainerStarted","Data":"7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e"} Nov 03 19:41:34.116098 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:34.115908 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" event={"ID":"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba","Type":"ContainerStarted","Data":"1ba2b5c1eb3b1fb9a9fb281d81158f0d88f99a887ab07f05bfcfdcc3a747c1e2"} Nov 03 19:41:34.116098 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:34.116054 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:34.116436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:34.116420 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:34.117044 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:34.117020 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:34.117136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:34.117067 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:34.137984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:34.137945 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" podStartSLOduration=2.137932959 podStartE2EDuration="2.137932959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:34.134882423 +0000 UTC m=+601.839721760" watchObservedRunningTime="2026-11-03 19:41:34.137932959 +0000 UTC m=+601.842772362" Nov 03 19:41:35.008545 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:35.008500 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:35.013921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:35.013899 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:35.122891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:35.122862 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:35.123877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:35.123853 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:41:37.026705 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.026674 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn"] Nov 03 19:41:37.030766 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.030747 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.037141 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.037117 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z"] Nov 03 19:41:37.039703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.039680 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-39043d-dockercfg-9jm99\"" Nov 03 19:41:37.039819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.039763 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-39043d\"" Nov 03 19:41:37.041597 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.041567 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.042336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.042317 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn"] Nov 03 19:41:37.058616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.047884 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z"] Nov 03 19:41:37.076771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.076731 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fljk\" (UniqueName: \"kubernetes.io/projected/f838cc11-3e00-41af-9a21-661befef36e3-kube-api-access-9fljk\") pod \"payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.076892 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.076774 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838cc11-3e00-41af-9a21-661befef36e3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.076892 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.076872 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55db36-642a-4799-a476-a15360e67526-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.077022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.076930 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnksh\" (UniqueName: \"kubernetes.io/projected/8e55db36-642a-4799-a476-a15360e67526-kube-api-access-bnksh\") pod \"payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.177890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.177849 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55db36-642a-4799-a476-a15360e67526-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.178095 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.177901 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnksh\" (UniqueName: \"kubernetes.io/projected/8e55db36-642a-4799-a476-a15360e67526-kube-api-access-bnksh\") pod \"payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.178095 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.178067 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fljk\" (UniqueName: \"kubernetes.io/projected/f838cc11-3e00-41af-9a21-661befef36e3-kube-api-access-9fljk\") pod \"payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.178226 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.178170 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838cc11-3e00-41af-9a21-661befef36e3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.178737 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.178717 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55db36-642a-4799-a476-a15360e67526-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.178926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.178908 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838cc11-3e00-41af-9a21-661befef36e3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.186151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.186126 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnksh\" (UniqueName: \"kubernetes.io/projected/8e55db36-642a-4799-a476-a15360e67526-kube-api-access-bnksh\") pod \"payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.186248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.186179 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fljk\" (UniqueName: \"kubernetes.io/projected/f838cc11-3e00-41af-9a21-661befef36e3-kube-api-access-9fljk\") pod \"payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.356520 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.356450 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:37.360851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.360832 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:37.641296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.641274 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn"] Nov 03 19:41:37.650807 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:37.650709 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf838cc11_3e00_41af_9a21_661befef36e3.slice/crio-b6567ae04933db7780d3dd4f1f79ab4ff7358ea2d107bd1a0441abb8f164ced3 WatchSource:0}: Error finding container b6567ae04933db7780d3dd4f1f79ab4ff7358ea2d107bd1a0441abb8f164ced3: Status 404 returned error can't find the container with id b6567ae04933db7780d3dd4f1f79ab4ff7358ea2d107bd1a0441abb8f164ced3 Nov 03 19:41:37.654126 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.654102 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z"] Nov 03 19:41:37.739891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.739860 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq"] Nov 03 19:41:37.743601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.743569 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:37.756607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.753840 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq"] Nov 03 19:41:37.797615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.795842 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a35658\"" Nov 03 19:41:37.811949 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.811919 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a35658-dockercfg-bzh4s\"" Nov 03 19:41:37.814218 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.814176 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a19d0b0-f577-4d5b-b503-22132c13e6ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:37.814537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.814515 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt445\" (UniqueName: \"kubernetes.io/projected/4a19d0b0-f577-4d5b-b503-22132c13e6ed-kube-api-access-mt445\") pod \"payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:37.915714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.915629 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a19d0b0-f577-4d5b-b503-22132c13e6ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:37.915714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.915699 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt445\" (UniqueName: \"kubernetes.io/projected/4a19d0b0-f577-4d5b-b503-22132c13e6ed-kube-api-access-mt445\") pod \"payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:37.916412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.916390 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a19d0b0-f577-4d5b-b503-22132c13e6ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:37.923755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:37.923735 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt445\" (UniqueName: \"kubernetes.io/projected/4a19d0b0-f577-4d5b-b503-22132c13e6ed-kube-api-access-mt445\") pod \"payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:38.064829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.064797 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:38.109127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.109049 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7"] Nov 03 19:41:38.116247 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.116223 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.135631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.128461 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7"] Nov 03 19:41:38.135631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.133203 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-39043d-serving-cert\"" Nov 03 19:41:38.143463 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.143431 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" event={"ID":"f838cc11-3e00-41af-9a21-661befef36e3","Type":"ContainerStarted","Data":"211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3"} Nov 03 19:41:38.143605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.143492 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" event={"ID":"f838cc11-3e00-41af-9a21-661befef36e3","Type":"ContainerStarted","Data":"b6567ae04933db7780d3dd4f1f79ab4ff7358ea2d107bd1a0441abb8f164ced3"} Nov 03 19:41:38.143605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.143524 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:38.147754 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.147724 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" event={"ID":"8e55db36-642a-4799-a476-a15360e67526","Type":"ContainerStarted","Data":"7c03cf39c1419594627414ed248977e9e350fd75f75058083334807c2a5361d2"} Nov 03 19:41:38.147878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.147776 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" event={"ID":"8e55db36-642a-4799-a476-a15360e67526","Type":"ContainerStarted","Data":"ad729c3f6b6c5b224e82606ee82d97e33d91f0ad9ff78af91d2ac9bee9a39b76"} Nov 03 19:41:38.148029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.147995 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:38.166870 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.166792 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" podStartSLOduration=1.166778175 podStartE2EDuration="1.166778175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:38.164194532 +0000 UTC m=+605.869033875" watchObservedRunningTime="2026-11-03 19:41:38.166778175 +0000 UTC m=+605.871617516" Nov 03 19:41:38.184546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.184505 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" podStartSLOduration=1.184491923 podStartE2EDuration="1.184491923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:38.182883341 +0000 UTC m=+605.887722682" watchObservedRunningTime="2026-11-03 19:41:38.184491923 +0000 UTC m=+605.889331263" Nov 03 19:41:38.237305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.237272 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-metrics-certs\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.237450 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.237318 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49683a7-346c-4ce5-8710-699b657febff-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.237450 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.237434 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-maas-api-tls\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.237577 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.237563 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqc4p\" (UniqueName: \"kubernetes.io/projected/b49683a7-346c-4ce5-8710-699b657febff-kube-api-access-xqc4p\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.339131 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.339094 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc4p\" (UniqueName: \"kubernetes.io/projected/b49683a7-346c-4ce5-8710-699b657febff-kube-api-access-xqc4p\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.339305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.339183 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-metrics-certs\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.339305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.339215 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49683a7-346c-4ce5-8710-699b657febff-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.339305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.339263 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-maas-api-tls\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.340083 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.340047 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49683a7-346c-4ce5-8710-699b657febff-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.341709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.341685 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-metrics-certs\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.341818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.341687 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-maas-api-tls\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.347151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.347127 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc4p\" (UniqueName: \"kubernetes.io/projected/b49683a7-346c-4ce5-8710-699b657febff-kube-api-access-xqc4p\") pod \"maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.354692 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:38.354669 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a19d0b0_f577_4d5b_b503_22132c13e6ed.slice/crio-e1080740cfdf042ba849bd1c1e5c9d0cfe74661fb3d5970d6f0ce946882b761b WatchSource:0}: Error finding container e1080740cfdf042ba849bd1c1e5c9d0cfe74661fb3d5970d6f0ce946882b761b: Status 404 returned error can't find the container with id e1080740cfdf042ba849bd1c1e5c9d0cfe74661fb3d5970d6f0ce946882b761b Nov 03 19:41:38.356280 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.356258 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq"] Nov 03 19:41:38.440279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.440252 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:38.721524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.721495 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7"] Nov 03 19:41:38.723784 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:38.723757 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49683a7_346c_4ce5_8710_699b657febff.slice/crio-3abf12695219fc1b7004ef0266e226bbc0f073a6e3984fb345798dfe001c64b4 WatchSource:0}: Error finding container 3abf12695219fc1b7004ef0266e226bbc0f073a6e3984fb345798dfe001c64b4: Status 404 returned error can't find the container with id 3abf12695219fc1b7004ef0266e226bbc0f073a6e3984fb345798dfe001c64b4 Nov 03 19:41:38.725438 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:38.725418 4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:41:39.160174 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.160143 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" event={"ID":"4a19d0b0-f577-4d5b-b503-22132c13e6ed","Type":"ContainerStarted","Data":"19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb"} Nov 03 19:41:39.160742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.160724 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:39.160866 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.160842 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" event={"ID":"4a19d0b0-f577-4d5b-b503-22132c13e6ed","Type":"ContainerStarted","Data":"e1080740cfdf042ba849bd1c1e5c9d0cfe74661fb3d5970d6f0ce946882b761b"} Nov 03 19:41:39.161745 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.161722 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" event={"ID":"b49683a7-346c-4ce5-8710-699b657febff","Type":"ContainerStarted","Data":"3abf12695219fc1b7004ef0266e226bbc0f073a6e3984fb345798dfe001c64b4"} Nov 03 19:41:39.182196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.182121 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" podStartSLOduration=2.182101734 podStartE2EDuration="2.182101734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:39.178526291 +0000 UTC m=+606.883365632" watchObservedRunningTime="2026-11-03 19:41:39.182101734 +0000 UTC m=+606.886941075" Nov 03 19:41:39.715935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.715899 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn"] Nov 03 19:41:39.720446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.720273 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:39.734117 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.734089 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn"] Nov 03 19:41:39.758397 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.758364 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ccfa05-c339-47bf-a71f-32fe3429410b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:39.758555 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.758530 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmthq\" (UniqueName: \"kubernetes.io/projected/93ccfa05-c339-47bf-a71f-32fe3429410b-kube-api-access-bmthq\") pod \"payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:39.779757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.779730 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-82dbbf\"" Nov 03 19:41:39.812237 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.812213 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-82dbbf-dockercfg-67nc6\"" Nov 03 19:41:39.861508 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.861439 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ccfa05-c339-47bf-a71f-32fe3429410b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:39.861939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.861914 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmthq\" (UniqueName: \"kubernetes.io/projected/93ccfa05-c339-47bf-a71f-32fe3429410b-kube-api-access-bmthq\") pod \"payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:39.863525 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.863437 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ccfa05-c339-47bf-a71f-32fe3429410b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:39.886600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:39.874231 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmthq\" (UniqueName: \"kubernetes.io/projected/93ccfa05-c339-47bf-a71f-32fe3429410b-kube-api-access-bmthq\") pod \"payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:40.039273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.039236 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds"] Nov 03 19:41:40.043246 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.043214 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.046926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.046902 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:40.055789 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.055771 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f4e123-serving-cert\"" Nov 03 19:41:40.061070 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.059660 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds"] Nov 03 19:41:40.165598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.165554 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-metrics-certs\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.166016 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.165626 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p7z\" (UniqueName: \"kubernetes.io/projected/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-kube-api-access-p9p7z\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.166016 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.165773 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.166016 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.165885 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.169641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.169612 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" event={"ID":"b49683a7-346c-4ce5-8710-699b657febff","Type":"ContainerStarted","Data":"c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657"} Nov 03 19:41:40.190773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.190705 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" podStartSLOduration=1.687198767 podStartE2EDuration="2.190685156s" totalImagesPullingTime="503.486389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:40.18831561 +0000 UTC m=+607.893154950" watchObservedRunningTime="2026-11-03 19:41:40.190685156 +0000 UTC m=+607.895524497" Nov 03 19:41:40.267572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.267537 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-metrics-certs\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.267794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.267608 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p7z\" (UniqueName: \"kubernetes.io/projected/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-kube-api-access-p9p7z\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.267794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.267636 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.267939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.267884 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.268463 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.268434 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.270348 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.270321 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-metrics-certs\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.270432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.270401 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.276040 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.276014 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p7z\" (UniqueName: \"kubernetes.io/projected/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-kube-api-access-p9p7z\") pod \"maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.337906 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.337800 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn"] Nov 03 19:41:40.341045 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:40.341005 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ccfa05_c339_47bf_a71f_32fe3429410b.slice/crio-8d521fc5e80cbb1d0b964c9577ea9a5180e96dcf7e4d4ef8e46e84f3b6a9e13e WatchSource:0}: Error finding container 8d521fc5e80cbb1d0b964c9577ea9a5180e96dcf7e4d4ef8e46e84f3b6a9e13e: Status 404 returned error can't find the container with id 8d521fc5e80cbb1d0b964c9577ea9a5180e96dcf7e4d4ef8e46e84f3b6a9e13e Nov 03 19:41:40.359309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.359061 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:40.670788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:40.670759 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds"] Nov 03 19:41:40.671515 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:40.671490 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706f41ed_2d34_46e8_8509_0bb2ff2e9f68.slice/crio-836f52fa5939fcab66440a3c2a9aa2543d2e739ce0e24205c73a540fcab7f8c7 WatchSource:0}: Error finding container 836f52fa5939fcab66440a3c2a9aa2543d2e739ce0e24205c73a540fcab7f8c7: Status 404 returned error can't find the container with id 836f52fa5939fcab66440a3c2a9aa2543d2e739ce0e24205c73a540fcab7f8c7 Nov 03 19:41:41.148623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.148564 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:41:41.180956 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.180922 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" event={"ID":"706f41ed-2d34-46e8-8509-0bb2ff2e9f68","Type":"ContainerStarted","Data":"836f52fa5939fcab66440a3c2a9aa2543d2e739ce0e24205c73a540fcab7f8c7"} Nov 03 19:41:41.183193 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.183166 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" event={"ID":"93ccfa05-c339-47bf-a71f-32fe3429410b","Type":"ContainerStarted","Data":"0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be"} Nov 03 19:41:41.183302 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.183244 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:41.183375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.183358 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:41.183429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.183382 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" event={"ID":"93ccfa05-c339-47bf-a71f-32fe3429410b","Type":"ContainerStarted","Data":"8d521fc5e80cbb1d0b964c9577ea9a5180e96dcf7e4d4ef8e46e84f3b6a9e13e"} Nov 03 19:41:41.203993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.203951 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" podStartSLOduration=2.20393684 podStartE2EDuration="2.20393684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:41.203454457 +0000 UTC m=+608.908293797" watchObservedRunningTime="2026-11-03 19:41:41.20393684 +0000 UTC m=+608.908776180" Nov 03 19:41:41.823431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.823393 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj"] Nov 03 19:41:41.828395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.828362 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.832748 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.832725 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-8d4127-serving-cert\"" Nov 03 19:41:41.835725 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.835439 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj"] Nov 03 19:41:41.887700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.887662 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.887880 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.887853 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.887967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.887951 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hwx\" (UniqueName: \"kubernetes.io/projected/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-kube-api-access-l9hwx\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.888026 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.887996 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-metrics-certs\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.989678 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.989636 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.989888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.989725 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hwx\" (UniqueName: \"kubernetes.io/projected/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-kube-api-access-l9hwx\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.989888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.989780 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-metrics-certs\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.989888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.989831 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.990667 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.990641 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.993240 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.993215 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.993373 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.993355 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-metrics-certs\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:41.998989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:41.998965 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hwx\" (UniqueName: \"kubernetes.io/projected/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-kube-api-access-l9hwx\") pod \"maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:42.148510 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:42.148432 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:42.192618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:42.191065 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" event={"ID":"706f41ed-2d34-46e8-8509-0bb2ff2e9f68","Type":"ContainerStarted","Data":"0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f"} Nov 03 19:41:42.192618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:42.191490 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:42.218431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:42.218384 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" podStartSLOduration=1.770481326 podStartE2EDuration="2.218364679s" totalImagesPullingTime="447.883353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:42.211830879 +0000 UTC m=+609.916670237" watchObservedRunningTime="2026-11-03 19:41:42.218364679 +0000 UTC m=+609.923204021" Nov 03 19:41:42.435509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:42.435479 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj"] Nov 03 19:41:42.437032 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:41:42.436992 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994e6de3_a05c_4cdd_b5b5_e753e2aeece0.slice/crio-25a69ccaa0cc0305c7828d2f8d0e53b613b844297779864c13d63d579d271594 WatchSource:0}: Error finding container 25a69ccaa0cc0305c7828d2f8d0e53b613b844297779864c13d63d579d271594: Status 404 returned error can't find the container with id 25a69ccaa0cc0305c7828d2f8d0e53b613b844297779864c13d63d579d271594 Nov 03 19:41:43.213531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:43.213347 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" event={"ID":"994e6de3-a05c-4cdd-b5b5-e753e2aeece0","Type":"ContainerStarted","Data":"e07f595423e13506e1d08836ae500ce59f6085ac60caaab44aa00b1ff46ded84"} Nov 03 19:41:43.213531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:43.213425 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" event={"ID":"994e6de3-a05c-4cdd-b5b5-e753e2aeece0","Type":"ContainerStarted","Data":"25a69ccaa0cc0305c7828d2f8d0e53b613b844297779864c13d63d579d271594"} Nov 03 19:41:43.214234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:43.214171 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:43.235250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:43.234491 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" podStartSLOduration=1.8190900669999999 podStartE2EDuration="2.234469011s" totalImagesPullingTime="415.378944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:43.230838773 +0000 UTC m=+610.935678091" watchObservedRunningTime="2026-11-03 19:41:43.234469011 +0000 UTC m=+610.939308354" Nov 03 19:41:45.527158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.527123 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb"] Nov 03 19:41:45.533728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.533704 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.538756 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.537665 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-102379-serving-cert\"" Nov 03 19:41:45.538756 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.538682 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb"] Nov 03 19:41:45.613605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.613099 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-maas-api-tls\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.613605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.613196 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-metrics-certs\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.613605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.613280 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbpj\" (UniqueName: \"kubernetes.io/projected/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-kube-api-access-4hbpj\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.613605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.613512 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.721610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.719959 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-maas-api-tls\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.721610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.720060 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-metrics-certs\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.721610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.720157 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbpj\" (UniqueName: \"kubernetes.io/projected/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-kube-api-access-4hbpj\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.721610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.720291 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.721610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.721311 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.725829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.725291 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-maas-api-tls\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.727479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.727452 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-metrics-certs\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.734521 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.732114 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbpj\" (UniqueName: \"kubernetes.io/projected/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-kube-api-access-4hbpj\") pod \"maas-api-e2e-worker-w6-102379-65549b87c-wzvpb\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:45.853004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:45.852926 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:46.282878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:46.282844 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb"] Nov 03 19:41:47.213683 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:47.212787 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 19:41:47.296605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:47.282318 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" event={"ID":"a33d9e6b-a0d3-491e-b41b-682cfe814fb4","Type":"ContainerStarted","Data":"24607051607b14a9afc529ad512d12762fb2d391308f7d5431632adfdc9c1787"} Nov 03 19:41:47.296605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:47.282383 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" event={"ID":"a33d9e6b-a0d3-491e-b41b-682cfe814fb4","Type":"ContainerStarted","Data":"2e8ed197649c8f597d1c7174ab3a9470881681cd79e34a7129ea1439613d0655"} Nov 03 19:41:47.296605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:47.283288 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:47.338649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:47.338592 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" podStartSLOduration=1.843714087 podStartE2EDuration="2.33856177s" totalImagesPullingTime="494.847683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:47.337498522 +0000 UTC m=+615.042337863" watchObservedRunningTime="2026-11-03 19:41:47.33856177 +0000 UTC m=+615.043401109" Nov 03 19:41:49.165675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:49.165645 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 19:41:49.166989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:49.166256 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 19:41:49.228993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:49.228959 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 19:41:49.229153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:49.229071 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 19:41:50.157783 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:50.157740 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:41:50.157951 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:50.157806 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:50.157951 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:50.157744 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:41:50.157951 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:50.157882 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:41:50.173785 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:50.173731 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 19:41:51.264428 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:51.264386 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:41:51.991771 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:41:51.991738 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:41:52.194701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.194625 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 19:41:52.321739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.321705 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" event={"ID":"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9","Type":"ContainerDied","Data":"bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51"} Nov 03 19:41:52.322237 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.321672 4314 generic.go:356] "Generic (PLEG): container finished" podID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerID="bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51" exitCode=0 Nov 03 19:41:52.322554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.322533 4314 scope.go:122] "RemoveContainer" containerID="bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51" Nov 03 19:41:52.324629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.324594 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:52.324758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.324676 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:52.324758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.324708 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:52.326893 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.326869 4314 generic.go:356] "Generic (PLEG): container finished" podID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerID="5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d" exitCode=0 Nov 03 19:41:52.327064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.326901 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" event={"ID":"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1","Type":"ContainerDied","Data":"5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d"} Nov 03 19:41:52.327518 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.327381 4314 scope.go:122] "RemoveContainer" containerID="5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d" Nov 03 19:41:52.328798 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.328765 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:52.328867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.328842 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:52.328903 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:52.328886 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:41:53.355446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.355410 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" event={"ID":"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9","Type":"ContainerStarted","Data":"50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9"} Nov 03 19:41:53.358953 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.358905 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" event={"ID":"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1","Type":"ContainerStarted","Data":"137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66"} Nov 03 19:41:53.504800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.504762 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:53.505406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.505293 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 19:41:53.505672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.505637 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 19:41:53.571164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.571128 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:53.571496 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.571472 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 19:41:53.571679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:53.571653 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 19:41:54.324621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:54.320699 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 19:41:54.626660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:54.624301 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 19:41:54.626660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:54.624360 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 19:41:54.626660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:54.626246 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 19:41:54.626660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:54.626289 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 19:41:55.504606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:55.504036 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 19:41:55.504606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:55.504099 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 19:41:55.571250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:55.571215 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 19:41:55.571600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:55.571543 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 19:41:56.507604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:56.503078 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 19:41:56.507604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:56.503154 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 19:41:56.565869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:56.565829 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 19:41:56.566941 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:56.566906 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 19:41:57.504001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:57.503955 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 19:41:57.504398 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:57.504367 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 19:41:57.569606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:57.565994 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 19:41:57.569606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:57.566055 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 19:41:58.510606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:58.510267 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 19:41:58.510606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:58.510328 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 19:41:58.568030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:58.567988 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 19:41:58.568318 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:58.568281 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 19:41:59.506159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:59.506127 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:59.506879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:59.506851 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:59.507652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:59.507620 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 19:41:59.574619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:59.573066 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:59.574619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:59.574237 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:59.575240 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:59.575023 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:41:59.730281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:41:59.730243 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:42:02.522984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:02.522951 4314 generic.go:356] "Generic (PLEG): container finished" podID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerID="7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e" exitCode=0 Nov 03 19:42:02.523392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:02.523017 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" event={"ID":"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba","Type":"ContainerDied","Data":"7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e"} Nov 03 19:42:02.523392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:02.523348 4314 scope.go:122] "RemoveContainer" containerID="7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e" Nov 03 19:42:02.524695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:02.524667 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:42:02.524773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:02.524736 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:42:02.524773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:02.524767 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:42:03.531083 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:03.531050 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" event={"ID":"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba","Type":"ContainerStarted","Data":"ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107"} Nov 03 19:42:04.008162 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:04.008124 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:42:04.009486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:04.009458 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 19:42:04.009626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:04.009507 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 19:42:05.008900 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:05.008864 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 19:42:05.009328 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:05.008926 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 19:42:05.123613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:05.123573 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:42:06.008715 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:06.008682 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 19:42:06.008916 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:06.008737 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 19:42:07.008560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:07.008235 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 19:42:07.008560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:07.008279 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 19:42:07.584683 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:07.582024 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:42:08.009405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:08.009367 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 19:42:08.009736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:08.009689 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 19:42:09.012810 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:09.012780 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:42:09.013690 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:09.013670 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 19:42:10.024401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.024366 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4"] Nov 03 19:42:10.027935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.027914 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.034228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.034205 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"simulator-w1-a35658-kserve-self-signed-certs\"" Nov 03 19:42:10.034431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.034405 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"kube-root-ca.crt\"" Nov 03 19:42:10.034507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.034318 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"openshift-service-ca.crt\"" Nov 03 19:42:10.034766 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.034744 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"default-dockercfg-s6mct\"" Nov 03 19:42:10.035961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.035937 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4"] Nov 03 19:42:10.072230 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.072200 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tls-certs\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.072386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.072262 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-home\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.072386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.072298 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-model-cache\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.072386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.072342 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247xq\" (UniqueName: \"kubernetes.io/projected/18d337f6-4a62-4fa7-a6ec-25f562474a8d-kube-api-access-247xq\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.072507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.072456 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-dshm\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.072616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.072598 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tmp-dir\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173441 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-home\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173475 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-model-cache\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173494 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247xq\" (UniqueName: \"kubernetes.io/projected/18d337f6-4a62-4fa7-a6ec-25f562474a8d-kube-api-access-247xq\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173538 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-dshm\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173641 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tmp-dir\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173731 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tls-certs\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173878 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-home\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.173927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.173903 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-model-cache\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.174020 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.174003 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tmp-dir\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.176073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.176051 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-dshm\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.176375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.176350 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tls-certs\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.182910 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.182885 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-247xq\" (UniqueName: \"kubernetes.io/projected/18d337f6-4a62-4fa7-a6ec-25f562474a8d-kube-api-access-247xq\") pod \"simulator-w1-a35658-kserve-5d4df55d8b-flst4\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.344698 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.344601 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:10.623531 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:10.623503 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4"] Nov 03 19:42:10.625631 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:42:10.625603 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d337f6_4a62_4fa7_a6ec_25f562474a8d.slice/crio-3c8865989bb0ebff548fdf74c520413e087f71d21f00701f1fb2adfe414d17fd WatchSource:0}: Error finding container 3c8865989bb0ebff548fdf74c520413e087f71d21f00701f1fb2adfe414d17fd: Status 404 returned error can't find the container with id 3c8865989bb0ebff548fdf74c520413e087f71d21f00701f1fb2adfe414d17fd Nov 03 19:42:11.579382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:11.579346 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" event={"ID":"18d337f6-4a62-4fa7-a6ec-25f562474a8d","Type":"ContainerStarted","Data":"c5596541f038c6c7e2d26a43d71b947eb83e4581ea7a43338cbece418936489e"} Nov 03 19:42:11.579382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:11.579388 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" event={"ID":"18d337f6-4a62-4fa7-a6ec-25f562474a8d","Type":"ContainerStarted","Data":"3c8865989bb0ebff548fdf74c520413e087f71d21f00701f1fb2adfe414d17fd"} Nov 03 19:42:11.596630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:11.596574 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podStartSLOduration=1.596561539 podStartE2EDuration="1.596561539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:11.595193143 +0000 UTC m=+639.300032482" watchObservedRunningTime="2026-11-03 19:42:11.596561539 +0000 UTC m=+639.301400882" Nov 03 19:42:12.345384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:12.345351 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:20.344959 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:20.344919 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:20.357846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:20.357815 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:20.646396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:20.646308 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:42:21.353350 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:21.353309 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:42:21.992268 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:42:21.992220 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:42:23.975089 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:23.975047 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt"] Nov 03 19:42:23.980868 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:23.980845 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:23.991637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:23.991572 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"premium-w2-8d4127-kserve-self-signed-certs\"" Nov 03 19:42:23.991894 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:23.991875 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"kube-root-ca.crt\"" Nov 03 19:42:23.992081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:23.992060 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"openshift-service-ca.crt\"" Nov 03 19:42:23.992154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:23.992138 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"default-dockercfg-2rgsc\"" Nov 03 19:42:24.007344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.007320 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt"] Nov 03 19:42:24.125384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.125353 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-tmp-dir\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.125554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.125393 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-model-cache\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.125554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.125520 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-dshm\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.125658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.125567 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-home\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.125699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.125657 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257d4457-e428-4e46-997d-dbab9754eb5d-tls-certs\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.125744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.125697 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sgd\" (UniqueName: \"kubernetes.io/projected/257d4457-e428-4e46-997d-dbab9754eb5d-kube-api-access-f8sgd\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227181 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227095 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-tmp-dir\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227181 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227140 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-model-cache\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227403 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227212 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-dshm\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227403 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227256 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-home\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227403 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227313 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257d4457-e428-4e46-997d-dbab9754eb5d-tls-certs\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227403 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227330 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sgd\" (UniqueName: \"kubernetes.io/projected/257d4457-e428-4e46-997d-dbab9754eb5d-kube-api-access-f8sgd\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227524 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-tmp-dir\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227634 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-home\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.227807 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.227787 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-model-cache\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.229479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.229459 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-dshm\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.230042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.230022 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257d4457-e428-4e46-997d-dbab9754eb5d-tls-certs\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.235254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.235229 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sgd\" (UniqueName: \"kubernetes.io/projected/257d4457-e428-4e46-997d-dbab9754eb5d-kube-api-access-f8sgd\") pod \"premium-w2-8d4127-kserve-b9bb7d448-cjhnt\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.297212 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.297190 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:24.572984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.572951 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt"] Nov 03 19:42:24.575325 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:42:24.575297 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod257d4457_e428_4e46_997d_dbab9754eb5d.slice/crio-9d4193086995009cc700f13a2a06664cb39b2335b46a4f86a9098c3c1b06c8f0 WatchSource:0}: Error finding container 9d4193086995009cc700f13a2a06664cb39b2335b46a4f86a9098c3c1b06c8f0: Status 404 returned error can't find the container with id 9d4193086995009cc700f13a2a06664cb39b2335b46a4f86a9098c3c1b06c8f0 Nov 03 19:42:24.659846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.659803 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" event={"ID":"257d4457-e428-4e46-997d-dbab9754eb5d","Type":"ContainerStarted","Data":"37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20"} Nov 03 19:42:24.659955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.659849 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" event={"ID":"257d4457-e428-4e46-997d-dbab9754eb5d","Type":"ContainerStarted","Data":"9d4193086995009cc700f13a2a06664cb39b2335b46a4f86a9098c3c1b06c8f0"} Nov 03 19:42:24.677489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:24.677448 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podStartSLOduration=1.67743397 podStartE2EDuration="1.67743397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:24.67650436 +0000 UTC m=+652.381343701" watchObservedRunningTime="2026-11-03 19:42:24.67743397 +0000 UTC m=+652.382273310" Nov 03 19:42:25.298108 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:25.298075 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:32.967271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:32.967235 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p"] Nov 03 19:42:32.972274 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:32.972252 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:32.980860 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:32.980835 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p"] Nov 03 19:42:32.984468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:32.984441 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"openshift-service-ca.crt\"" Nov 03 19:42:32.984770 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:32.984745 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"kube-root-ca.crt\"" Nov 03 19:42:32.985491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:32.985471 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"premium-w3-39043d-kserve-self-signed-certs\"" Nov 03 19:42:32.985668 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:32.985614 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"default-dockercfg-df4p9\"" Nov 03 19:42:33.031182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.031159 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-model-cache\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.031182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.031190 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-dshm\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.031350 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.031210 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tmp-dir\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.031350 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.031296 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjgc\" (UniqueName: \"kubernetes.io/projected/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-kube-api-access-4hjgc\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.031350 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.031327 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tls-certs\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.031443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.031362 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-home\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.132861 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.132811 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-home\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.133069 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.132973 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-model-cache\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.133069 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.133007 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-dshm\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.133069 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.133034 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tmp-dir\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.133209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.133075 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjgc\" (UniqueName: \"kubernetes.io/projected/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-kube-api-access-4hjgc\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.133209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.133119 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tls-certs\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.135351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.134313 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-home\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.135351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.134617 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-model-cache\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.135351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.134982 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tmp-dir\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.136273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.136253 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-dshm\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.136843 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.136772 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tls-certs\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.143486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.143461 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjgc\" (UniqueName: \"kubernetes.io/projected/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-kube-api-access-4hjgc\") pod \"premium-w3-39043d-kserve-846bcd8466-sls7p\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.293755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.293720 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:33.584258 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.584228 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p"] Nov 03 19:42:33.586571 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:42:33.586539 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bb057b6_7660_4a8e_9cd5_3ffe086cd72d.slice/crio-4a30e7c6d8868de704e6666aa719039703d4b52967cb208596f5fd02022ad2cd WatchSource:0}: Error finding container 4a30e7c6d8868de704e6666aa719039703d4b52967cb208596f5fd02022ad2cd: Status 404 returned error can't find the container with id 4a30e7c6d8868de704e6666aa719039703d4b52967cb208596f5fd02022ad2cd Nov 03 19:42:33.721848 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.721817 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" event={"ID":"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d","Type":"ContainerStarted","Data":"d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7"} Nov 03 19:42:33.722026 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.721860 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" event={"ID":"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d","Type":"ContainerStarted","Data":"4a30e7c6d8868de704e6666aa719039703d4b52967cb208596f5fd02022ad2cd"} Nov 03 19:42:33.737872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:33.737828 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podStartSLOduration=1.737815737 podStartE2EDuration="1.737815737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:33.737245683 +0000 UTC m=+661.442085017" watchObservedRunningTime="2026-11-03 19:42:33.737815737 +0000 UTC m=+661.442655076" Nov 03 19:42:34.294838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:34.294801 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:34.298085 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:34.298053 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:34.311857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:34.311830 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:34.352724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:34.352697 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:42:34.745392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:34.745361 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:42:40.065957 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.063503 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx"] Nov 03 19:42:40.069703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.069629 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.075357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.075328 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"kube-root-ca.crt\"" Nov 03 19:42:40.075482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.075451 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"premium-w6-102379-kserve-self-signed-certs\"" Nov 03 19:42:40.075482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.075453 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"openshift-service-ca.crt\"" Nov 03 19:42:40.077043 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.077016 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx"] Nov 03 19:42:40.122094 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.122065 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tls-certs\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.122253 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.122111 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tmp-dir\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.122253 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.122218 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-model-cache\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.122340 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.122265 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-dshm\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.122386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.122354 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-home\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.122423 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.122404 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zgb\" (UniqueName: \"kubernetes.io/projected/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-kube-api-access-d6zgb\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.173685 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.173650 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"default-dockercfg-cv6dk\"" Nov 03 19:42:40.223741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.223703 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-model-cache\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.223933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.223773 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-dshm\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.223933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.223825 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-home\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.223933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.223888 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zgb\" (UniqueName: \"kubernetes.io/projected/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-kube-api-access-d6zgb\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.224113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.223984 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tls-certs\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.224113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.224032 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tmp-dir\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.224252 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.224209 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-home\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.224322 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.224301 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-model-cache\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.224384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.224371 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tmp-dir\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.226148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.226125 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-dshm\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.226411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.226392 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tls-certs\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.231835 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.231812 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zgb\" (UniqueName: \"kubernetes.io/projected/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-kube-api-access-d6zgb\") pod \"premium-w6-102379-kserve-5678c74646-pnrpx\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.389271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.389175 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:40.676572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.676547 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx"] Nov 03 19:42:40.679131 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:42:40.679100 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda318f53f_7b2c_4b68_96e9_51fcd403c2d6.slice/crio-ef1a3fc899337af59208f1dc007fee762cb6985236aa173ee2eabc8254d9ccb7 WatchSource:0}: Error finding container ef1a3fc899337af59208f1dc007fee762cb6985236aa173ee2eabc8254d9ccb7: Status 404 returned error can't find the container with id ef1a3fc899337af59208f1dc007fee762cb6985236aa173ee2eabc8254d9ccb7 Nov 03 19:42:40.766532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.766498 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" event={"ID":"a318f53f-7b2c-4b68-96e9-51fcd403c2d6","Type":"ContainerStarted","Data":"96f2a89509b3780b0c37ae0457e5bb32c71f5bb4f7a176c208878bc14e55af3a"} Nov 03 19:42:40.766682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.766543 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" event={"ID":"a318f53f-7b2c-4b68-96e9-51fcd403c2d6","Type":"ContainerStarted","Data":"ef1a3fc899337af59208f1dc007fee762cb6985236aa173ee2eabc8254d9ccb7"} Nov 03 19:42:40.783043 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:40.782985 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podStartSLOduration=0.782968416 podStartE2EDuration="782.968416ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:40.782460052 +0000 UTC m=+668.487299397" watchObservedRunningTime="2026-11-03 19:42:40.782968416 +0000 UTC m=+668.487807756" Nov 03 19:42:41.390056 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:41.390021 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:43.294681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:43.294638 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:43.311417 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:43.311394 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:43.799287 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:43.799249 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:42:50.389797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:50.389705 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:50.407829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:50.407797 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:50.861325 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:50.861295 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:42:51.994515 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:42:51.994478 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:42:55.577448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:42:55.577407 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:43:02.639798 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.639762 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m"] Nov 03 19:43:02.645087 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.645068 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.651315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.651294 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f4e123-openshift-default-dockercfg-qlw2x\"" Nov 03 19:43:02.655088 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.655059 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m"] Nov 03 19:43:02.692792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.692758 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693072 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693044 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693165 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9c4\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-kube-api-access-fh9c4\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693246 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693343 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693288 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693353 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693462 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693506 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.693675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.693537 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.794709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.794667 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9c4\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-kube-api-access-fh9c4\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.794874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.794726 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.794874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.794755 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.794874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.794792 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.794874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.794833 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.794874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.794863 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.794890 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.795010 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.795081 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795288 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.795168 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795288 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.795202 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.795458 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795545 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.795478 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.795813 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.795792 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.797752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.797728 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.797896 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.797877 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.802869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.802831 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.803155 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.803138 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9c4\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-kube-api-access-fh9c4\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:02.970990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:02.970907 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:03.264336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.264310 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m"] Nov 03 19:43:03.267098 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:43:03.267072 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5706b8d_6931_4fcc_bd8a_06009abcebdc.slice/crio-b7ef96289a5973e40135844d7a1fef389657bdad477da740ebc4a68055302ab3 WatchSource:0}: Error finding container b7ef96289a5973e40135844d7a1fef389657bdad477da740ebc4a68055302ab3: Status 404 returned error can't find the container with id b7ef96289a5973e40135844d7a1fef389657bdad477da740ebc4a68055302ab3 Nov 03 19:43:03.270355 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.270312 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:03.270489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.270404 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:03.270489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.270450 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:03.933486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.933444 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" event={"ID":"e5706b8d-6931-4fcc-bd8a-06009abcebdc","Type":"ContainerStarted","Data":"21edf33b2d7ff1267a4b24cfac6dcd45d5073e1395328cb19f82b3fbacc02204"} Nov 03 19:43:03.933952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.933504 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" event={"ID":"e5706b8d-6931-4fcc-bd8a-06009abcebdc","Type":"ContainerStarted","Data":"b7ef96289a5973e40135844d7a1fef389657bdad477da740ebc4a68055302ab3"} Nov 03 19:43:03.960604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.959061 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podStartSLOduration=1.959044901 podStartE2EDuration="1.959044901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:03.954659043 +0000 UTC m=+691.659498384" watchObservedRunningTime="2026-11-03 19:43:03.959044901 +0000 UTC m=+691.663884245" Nov 03 19:43:03.978604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:03.975674 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:04.982723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:04.982680 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:43:04.983283 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:04.982752 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:43:05.988605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:05.986016 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:43:05.988605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:05.986095 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:43:06.972922 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:06.972885 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:43:06.973204 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:06.973177 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:43:07.975604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:07.974365 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:43:07.975604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:07.974439 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:43:08.978606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:08.977168 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:43:08.978606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:08.977222 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:43:09.976911 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:09.976874 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:09.977282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:09.977246 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:09.978352 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:09.978331 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 19:43:17.202670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.202565 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq"] Nov 03 19:43:17.207828 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.207807 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.217861 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.217819 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8d4127-openshift-default-dockercfg-6m7xm\"" Nov 03 19:43:17.218636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.218613 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq"] Nov 03 19:43:17.294228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294196 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294379 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294237 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294379 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294265 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294366 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294548 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294520 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294639 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsxn\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-kube-api-access-spsxn\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294737 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294720 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294776 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.294919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.294900 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.399227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.399184 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.400169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.400149 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.400673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.400655 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.401161 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.401142 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.401606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.401570 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsxn\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-kube-api-access-spsxn\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.402018 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.401998 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.402546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.402524 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.403000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.402980 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.403157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.403140 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.403295 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.402889 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.403387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.401492 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.403982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.400074 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.403982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.403940 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.412883 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.412861 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.416167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.416146 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsxn\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-kube-api-access-spsxn\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.419856 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.419836 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.426639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.422134 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.426800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.422559 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.525953 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.525915 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:17.820665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.820635 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq"] Nov 03 19:43:17.822136 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:43:17.822113 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb3b604_88e1_44b9_a4aa_9184c46708fa.slice/crio-241b1e2613ed896c70beb4795d4860a34ad253562e2b09927bf201e4f6a02cc2 WatchSource:0}: Error finding container 241b1e2613ed896c70beb4795d4860a34ad253562e2b09927bf201e4f6a02cc2: Status 404 returned error can't find the container with id 241b1e2613ed896c70beb4795d4860a34ad253562e2b09927bf201e4f6a02cc2 Nov 03 19:43:17.824831 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.824797 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:17.824904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.824865 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:17.824904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:17.824893 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:18.035416 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:18.035376 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" event={"ID":"3bb3b604-88e1-44b9-a4aa-9184c46708fa","Type":"ContainerStarted","Data":"119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5"} Nov 03 19:43:18.035629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:18.035442 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" event={"ID":"3bb3b604-88e1-44b9-a4aa-9184c46708fa","Type":"ContainerStarted","Data":"241b1e2613ed896c70beb4795d4860a34ad253562e2b09927bf201e4f6a02cc2"} Nov 03 19:43:18.084401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:18.084307 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podStartSLOduration=1.084293373 podStartE2EDuration="1.084293373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:18.081384993 +0000 UTC m=+705.786224327" watchObservedRunningTime="2026-11-03 19:43:18.084293373 +0000 UTC m=+705.789132713" Nov 03 19:43:18.526492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:18.526457 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:18.527674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:18.527651 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:43:18.527793 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:18.527697 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:43:19.526750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:19.526704 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:43:19.527261 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:19.526765 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:43:20.526430 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:20.526393 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:43:20.526633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:20.526447 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:43:21.526987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:21.526954 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:43:21.527491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:21.527018 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:43:22.025434 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:43:22.025375 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:43:22.526880 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:22.526850 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:43:22.527054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:22.526907 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:43:23.526699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:23.526666 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:43:23.526889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:23.526722 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:43:24.531034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:24.531005 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:24.531483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:24.531442 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:24.532073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:24.532052 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:43:27.447709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:27.447677 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:43:32.472747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.472715 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4"] Nov 03 19:43:32.477154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.477131 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.498604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.483949 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-82dbbf-openshift-default-dockercfg-wz5bm\"" Nov 03 19:43:32.521604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.509254 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4"] Nov 03 19:43:32.625882 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.625852 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.625906 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626150 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.626138 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngr9\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-kube-api-access-cngr9\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.626259 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.626304 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.626389 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626471 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.626441 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.626508 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.626798 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.626779 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.727860 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.727781 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngr9\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-kube-api-access-cngr9\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.727860 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.727850 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.727878 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.727930 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.727955 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.727990 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728036 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728205 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728255 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728368 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728372 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.728512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728476 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.729867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728747 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.729867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.728759 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.730883 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.730858 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.734646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.734477 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.736656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.736631 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.739403 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.739325 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngr9\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-kube-api-access-cngr9\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:32.836082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:32.836051 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:33.122939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:33.122912 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4"] Nov 03 19:43:33.125108 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:43:33.125076 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39f0f54_a9ca_4bdd_81ef_2c47e4d00c0f.slice/crio-0d4a24c9cbd99999eb5336fb34128334a58351fd3eee8926b79379cb05d70ef7 WatchSource:0}: Error finding container 0d4a24c9cbd99999eb5336fb34128334a58351fd3eee8926b79379cb05d70ef7: Status 404 returned error can't find the container with id 0d4a24c9cbd99999eb5336fb34128334a58351fd3eee8926b79379cb05d70ef7 Nov 03 19:43:33.128139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:33.128101 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:33.128249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:33.128175 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:33.128249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:33.128213 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:43:33.178859 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:33.178830 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" event={"ID":"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f","Type":"ContainerStarted","Data":"0d4a24c9cbd99999eb5336fb34128334a58351fd3eee8926b79379cb05d70ef7"} Nov 03 19:43:34.196511 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:34.196466 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" event={"ID":"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f","Type":"ContainerStarted","Data":"e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591"} Nov 03 19:43:34.218214 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:34.218162 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podStartSLOduration=2.2181471520000002 podStartE2EDuration="2.218147152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:34.215951281 +0000 UTC m=+721.920790638" watchObservedRunningTime="2026-11-03 19:43:34.218147152 +0000 UTC m=+721.922986492" Nov 03 19:43:34.836788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:34.836755 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:34.838619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:34.838572 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:43:34.838764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:34.838656 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:43:35.836335 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:35.836298 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:43:35.836790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:35.836361 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:43:36.836371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:36.836335 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:43:36.836797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:36.836393 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:43:37.836373 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:37.836337 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:43:37.836790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:37.836392 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:43:38.836776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:38.836741 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:43:38.837180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:38.836792 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:43:39.855505 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:39.855468 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:39.856747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:39.856725 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:39.857958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:43:39.857939 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:43:52.008959 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:43:52.008827 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:44:00.815108 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:00.815073 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:44:03.804728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:03.803674 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:44:17.433961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.433856 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r"] Nov 03 19:44:17.519050 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.519015 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.544401 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r"] Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.545979 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546016 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546087 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5vh\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-kube-api-access-ss5vh\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546121 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546196 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546225 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546317 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546374 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.559441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.546419 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657322 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657383 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657449 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657481 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657546 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5vh\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-kube-api-access-ss5vh\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657600 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657672 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657706 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.657806 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.658344 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.680053 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.680551 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.701684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.680831 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:17.736607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:17.711399 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:18.056628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.026408 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:18.082605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.058656 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:18.097605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.083768 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:18.135606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.097810 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22"] Nov 03 19:44:18.135606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.104122 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.135606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.126572 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22"] Nov 03 19:44:18.261400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.127569 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0486e8-openshift-default-dockercfg-4ckf8\"" Nov 03 19:44:18.261400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.178967 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5vh\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-kube-api-access-ss5vh\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:18.270603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.267428 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.288742 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.288860 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.288927 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.288951 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d72718-b839-4e03-a321-fdb818d157e7-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.289032 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d72718-b839-4e03-a321-fdb818d157e7-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.289090 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.289126 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.292604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.289247 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkt48\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-kube-api-access-qkt48\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409202 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409247 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409319 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt48\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-kube-api-access-qkt48\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409389 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409444 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409507 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409555 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409596 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d72718-b839-4e03-a321-fdb818d157e7-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.409648 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d72718-b839-4e03-a321-fdb818d157e7-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.411382 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.411839 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.416648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.412096 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.427001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.418721 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.427001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.422783 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.427001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.423258 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.427001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.423786 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d72718-b839-4e03-a321-fdb818d157e7-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.447527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.447240 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d72718-b839-4e03-a321-fdb818d157e7-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.464752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.464715 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt48\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-kube-api-access-qkt48\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.535743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.535716 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:18.606608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.598442 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:18.879229 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.879190 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r"] Nov 03 19:44:18.883250 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:44:18.883209 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d193b3d_3bf2_4b84_a6cc_7e101e9f4fe5.slice/crio-e93ca494f679381dc1e32b77d1298af6c9deba8e2ef83d8de453d08b62cdf772 WatchSource:0}: Error finding container e93ca494f679381dc1e32b77d1298af6c9deba8e2ef83d8de453d08b62cdf772: Status 404 returned error can't find the container with id e93ca494f679381dc1e32b77d1298af6c9deba8e2ef83d8de453d08b62cdf772 Nov 03 19:44:18.887625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.886841 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:18.887749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.887690 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:18.887749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.887743 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:18.933829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.933781 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22"] Nov 03 19:44:18.934887 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:44:18.934859 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d72718_b839_4e03_a321_fdb818d157e7.slice/crio-6a23b200d59f9821dc001c5812795d75ddf4bce7a1a6fafed77e1e22503cb38e WatchSource:0}: Error finding container 6a23b200d59f9821dc001c5812795d75ddf4bce7a1a6fafed77e1e22503cb38e: Status 404 returned error can't find the container with id 6a23b200d59f9821dc001c5812795d75ddf4bce7a1a6fafed77e1e22503cb38e Nov 03 19:44:18.937512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.937472 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:18.937608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.937572 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:18.937669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:18.937641 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:19.283155 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.283117 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" event={"ID":"62d72718-b839-4e03-a321-fdb818d157e7","Type":"ContainerStarted","Data":"4b89e380c76b7883561d33f35de7f494afdfd5eeb401f6d36ac6a0e3efa4b8b1"} Nov 03 19:44:19.283355 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.283188 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" event={"ID":"62d72718-b839-4e03-a321-fdb818d157e7","Type":"ContainerStarted","Data":"6a23b200d59f9821dc001c5812795d75ddf4bce7a1a6fafed77e1e22503cb38e"} Nov 03 19:44:19.285125 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.285094 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" event={"ID":"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5","Type":"ContainerStarted","Data":"5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c"} Nov 03 19:44:19.285232 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.285147 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" event={"ID":"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5","Type":"ContainerStarted","Data":"e93ca494f679381dc1e32b77d1298af6c9deba8e2ef83d8de453d08b62cdf772"} Nov 03 19:44:19.332032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.331645 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podStartSLOduration=2.331628331 podStartE2EDuration="2.331628331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:19.3310296 +0000 UTC m=+767.035868941" watchObservedRunningTime="2026-11-03 19:44:19.331628331 +0000 UTC m=+767.036467673" Nov 03 19:44:19.333727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.333684 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podStartSLOduration=2.3336759320000002 podStartE2EDuration="2.333675932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:19.308847442 +0000 UTC m=+767.013686807" watchObservedRunningTime="2026-11-03 19:44:19.333675932 +0000 UTC m=+767.038515272" Nov 03 19:44:19.536811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.536722 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:19.538303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.538275 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:44:19.538449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.538326 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:44:19.600512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:19.600463 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:20.536238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:20.536202 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:44:20.536412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:20.536256 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:44:20.612616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:20.609602 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:44:20.612616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:20.609697 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:44:21.536926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:21.536891 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:44:21.537163 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:21.537143 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:44:21.605001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:21.604947 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:44:21.605205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:21.605010 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:44:22.039376 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:44:22.039329 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:44:22.537879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:22.537841 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:44:22.538061 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:22.537889 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:44:22.607602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:22.606817 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:44:22.607602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:22.606876 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:44:23.460607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.456873 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb"] Nov 03 19:44:23.510605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.499327 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.528645 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.511107 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"distinct-w6-102379-kserve-self-signed-certs\"" Nov 03 19:44:23.539533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.536196 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:44:23.539533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.536248 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:44:23.542443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.542370 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb"] Nov 03 19:44:23.618687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.618042 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:44:23.618687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.618115 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:44:23.646613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.641764 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa79738-f4df-4b25-9319-ab36d813b7b9-tls-certs\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.646613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.641833 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-dshm\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.646613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.642030 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-tmp-dir\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.646613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.642209 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-home\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.646613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.642270 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-model-cache\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.646613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.642301 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfvc\" (UniqueName: \"kubernetes.io/projected/cfa79738-f4df-4b25-9319-ab36d813b7b9-kube-api-access-rcfvc\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.743141 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743102 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-dshm\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.743314 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743199 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-tmp-dir\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.743314 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743309 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-home\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.743393 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743366 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-model-cache\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.743436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743398 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfvc\" (UniqueName: \"kubernetes.io/projected/cfa79738-f4df-4b25-9319-ab36d813b7b9-kube-api-access-rcfvc\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.743498 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743482 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa79738-f4df-4b25-9319-ab36d813b7b9-tls-certs\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.743956 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743926 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-tmp-dir\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.744073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.743938 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-home\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.744073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.744002 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-model-cache\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.746097 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.746066 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-dshm\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.746396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.746379 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa79738-f4df-4b25-9319-ab36d813b7b9-tls-certs\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.752495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.752471 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfvc\" (UniqueName: \"kubernetes.io/projected/cfa79738-f4df-4b25-9319-ab36d813b7b9-kube-api-access-rcfvc\") pod \"distinct-w6-102379-kserve-78d65d6fb5-cxxnb\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:23.845524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:23.843070 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:24.329652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:24.328404 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb"] Nov 03 19:44:24.356442 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:24.356414 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" event={"ID":"cfa79738-f4df-4b25-9319-ab36d813b7b9","Type":"ContainerStarted","Data":"09ea4cc8018472fe8985371b66c8b1a54eebf54e40861c6d925c19a5bde83b6b"} Nov 03 19:44:24.536470 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:24.536091 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:44:24.536470 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:24.536144 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:44:24.607662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:24.606938 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:44:24.607662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:24.607001 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:44:25.367384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:25.367332 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" event={"ID":"cfa79738-f4df-4b25-9319-ab36d813b7b9","Type":"ContainerStarted","Data":"dd0519ace3664cc27ecbe80dd4c4b90dfe3075c1208123526a22985a43b40812"} Nov 03 19:44:25.389000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:25.388922 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podStartSLOduration=2.388903298 podStartE2EDuration="2.388903298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:25.386457861 +0000 UTC m=+773.091297202" watchObservedRunningTime="2026-11-03 19:44:25.388903298 +0000 UTC m=+773.093742641" Nov 03 19:44:25.537009 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:25.536970 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:44:25.537510 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:25.537038 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:44:25.600540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:25.600499 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:44:25.600741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:25.600559 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:44:25.843982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:25.843947 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:26.541165 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:26.541136 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:26.541693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:26.541659 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:26.542499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:26.542472 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 19:44:26.610102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:26.610065 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:26.610972 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:26.610944 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:26.614168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:26.613525 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:44:30.823293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:30.823259 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:44:33.023720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.022367 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6"] Nov 03 19:44:33.038608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.029668 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.048077 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.048047 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6"] Nov 03 19:44:33.129605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.117603 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk"] Nov 03 19:44:33.167605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.155978 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.169567 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.191934 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.191994 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.192061 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.192148 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.192202 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj7k\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-kube-api-access-klj7k\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.192242 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.192269 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.193561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.192384 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.309878 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.309976 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310021 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310058 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj7k\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-kube-api-access-klj7k\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310097 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310129 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310189 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310250 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310277 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310355 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4ks\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-kube-api-access-nf4ks\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310404 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310436 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310463 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767e45e0-6fdd-4f2e-a884-e335281262ba-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310495 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310521 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310563 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310614 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.310655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.310663 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.311692 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.311101 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.384607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.360359 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.394037 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.364430 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.394236 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.380819 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.405171 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.405139 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419430 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419564 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419651 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419731 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4ks\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-kube-api-access-nf4ks\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419787 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419822 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767e45e0-6fdd-4f2e-a884-e335281262ba-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419854 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419902 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.419982 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.420531 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.447620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.420817 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.451092 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.451066 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.452231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.452205 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.453067 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.453045 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767e45e0-6fdd-4f2e-a884-e335281262ba-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.457893 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.457870 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.479521 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.479492 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj7k\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-kube-api-access-klj7k\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.494232 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.493815 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk"] Nov 03 19:44:33.555674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.555114 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.673995 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.673912 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.694335 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.694301 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.713205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.713174 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4ks\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-kube-api-access-nf4ks\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:33.714128 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.714106 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.732703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.732663 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:44:33.740284 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.740260 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.759157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.758928 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:33.848689 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.843670 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:44:33.859842 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.858409 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:33.935800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:33.935426 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:34.148436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.143133 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:34.179486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.179426 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:44:34.336135 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.336103 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6"] Nov 03 19:44:34.340769 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.340724 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:34.340872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.340814 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:34.340872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.340859 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:34.431481 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:44:34.431452 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767e45e0_6fdd_4f2e_a884_e335281262ba.slice/crio-2e62440b58a11f66fc7b5d25b80ec5e5e24c96a9e073d1b0bec756f0b7cf74a2 WatchSource:0}: Error finding container 2e62440b58a11f66fc7b5d25b80ec5e5e24c96a9e073d1b0bec756f0b7cf74a2: Status 404 returned error can't find the container with id 2e62440b58a11f66fc7b5d25b80ec5e5e24c96a9e073d1b0bec756f0b7cf74a2 Nov 03 19:44:34.434038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.434005 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:34.434153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.434081 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:34.434153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.434111 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:44:34.436767 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.436746 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk"] Nov 03 19:44:34.878018 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.877933 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" event={"ID":"767e45e0-6fdd-4f2e-a884-e335281262ba","Type":"ContainerStarted","Data":"bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f"} Nov 03 19:44:34.878018 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.877990 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" event={"ID":"767e45e0-6fdd-4f2e-a884-e335281262ba","Type":"ContainerStarted","Data":"2e62440b58a11f66fc7b5d25b80ec5e5e24c96a9e073d1b0bec756f0b7cf74a2"} Nov 03 19:44:34.880228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.880201 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" event={"ID":"f553b82b-9c58-4eec-ac79-9915f42e7ae9","Type":"ContainerStarted","Data":"cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49"} Nov 03 19:44:34.880388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.880248 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" event={"ID":"f553b82b-9c58-4eec-ac79-9915f42e7ae9","Type":"ContainerStarted","Data":"7cb206517e62c2173ff6e2ccd67166ce187f472fc7e6c4e6f715ace23aef9a68"} Nov 03 19:44:34.900721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.900669 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podStartSLOduration=1.900656391 podStartE2EDuration="1.900656391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:34.897832905 +0000 UTC m=+782.602672245" watchObservedRunningTime="2026-11-03 19:44:34.900656391 +0000 UTC m=+782.605495730" Nov 03 19:44:34.922502 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.922439 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podStartSLOduration=2.922421713 podStartE2EDuration="2.922421713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:34.917670408 +0000 UTC m=+782.622509749" watchObservedRunningTime="2026-11-03 19:44:34.922421713 +0000 UTC m=+782.627261056" Nov 03 19:44:34.936060 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:34.936007 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:35.761878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:35.761846 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:35.766709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:35.766677 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:44:35.766954 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:35.766928 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:44:35.937220 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:35.937175 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 19:44:35.937424 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:35.937242 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 19:44:36.766605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:36.764870 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:44:36.766605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:36.764932 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:44:36.936100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:36.936063 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 19:44:36.936253 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:36.936114 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 19:44:37.761924 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:37.761889 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:44:37.762185 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:37.762159 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:44:37.936065 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:37.936027 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 19:44:37.936572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:37.936085 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 19:44:38.777079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:38.777041 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:44:38.777357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:38.777329 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:44:38.938038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:38.937993 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 19:44:38.938622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:38.938591 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 19:44:39.760898 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:39.760861 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:44:39.761190 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:39.761158 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:44:39.936060 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:39.936017 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 19:44:39.936237 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:39.936064 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 19:44:40.760632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:40.760438 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:44:40.760632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:40.760495 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:44:40.936228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:40.936188 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 19:44:40.936413 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:40.936258 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 19:44:41.765706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:41.765650 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:41.766192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:41.766020 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:41.766834 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:41.766810 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 19:44:41.940714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:41.940685 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:41.941165 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:41.941145 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:41.941883 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:41.941864 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:44:42.050891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.050807 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff"] Nov 03 19:44:42.056360 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.056335 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.063351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.063325 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"distinct-2-w1-a35658-kserve-self-signed-certs\"" Nov 03 19:44:42.069473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.068420 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff"] Nov 03 19:44:42.218076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.217653 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tmp-dir\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.218076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.217741 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tls-certs\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.218076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.217834 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-home\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.218076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.217863 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-model-cache\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.219021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.218731 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlvl\" (UniqueName: \"kubernetes.io/projected/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-kube-api-access-7dlvl\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.219021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.218819 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-dshm\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320200 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tmp-dir\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320263 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tls-certs\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320339 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-home\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320365 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-model-cache\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320488 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlvl\" (UniqueName: \"kubernetes.io/projected/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-kube-api-access-7dlvl\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320705 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320541 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-dshm\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320769 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tmp-dir\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320848 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-home\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.320914 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.320873 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-model-cache\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.322764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.322736 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-dshm\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.322863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.322797 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tls-certs\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.328137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.328114 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlvl\" (UniqueName: \"kubernetes.io/projected/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-kube-api-access-7dlvl\") pod \"distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.393298 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.393253 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:42.678315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:42.678283 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff"] Nov 03 19:44:42.680483 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:44:42.680454 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ff0d55c_5ae2_45be_8edb_d770f7ffb20c.slice/crio-0234943c6e531e900394dede9d561d2ae456393c4eb5916c8780688e76b15c3d WatchSource:0}: Error finding container 0234943c6e531e900394dede9d561d2ae456393c4eb5916c8780688e76b15c3d: Status 404 returned error can't find the container with id 0234943c6e531e900394dede9d561d2ae456393c4eb5916c8780688e76b15c3d Nov 03 19:44:43.009883 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:43.009850 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" event={"ID":"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c","Type":"ContainerStarted","Data":"494bf993d36d70742affa9b932b6232b310945e9c6d774c9aa44cd8afb3949bf"} Nov 03 19:44:43.010373 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:43.009893 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" event={"ID":"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c","Type":"ContainerStarted","Data":"0234943c6e531e900394dede9d561d2ae456393c4eb5916c8780688e76b15c3d"} Nov 03 19:44:43.028141 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:43.028089 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podStartSLOduration=1.02807545 podStartE2EDuration="1.02807545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:43.02596839 +0000 UTC m=+790.730807733" watchObservedRunningTime="2026-11-03 19:44:43.02807545 +0000 UTC m=+790.732914789" Nov 03 19:44:43.394547 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:43.394471 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:47.730544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.730503 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm"] Nov 03 19:44:47.743418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.743383 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm"] Nov 03 19:44:47.743726 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.743707 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.747795 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.747769 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"distinct-w3-39043d-kserve-self-signed-certs\"" Nov 03 19:44:47.832927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.832892 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-tmp-dir\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.833088 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.832945 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-dshm\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.833146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.833073 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdjm\" (UniqueName: \"kubernetes.io/projected/06bf6477-c859-4d11-8de8-aff72c7d849d-kube-api-access-9hdjm\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.833285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.833264 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-home\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.833344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.833295 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bf6477-c859-4d11-8de8-aff72c7d849d-tls-certs\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.833401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.833385 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-model-cache\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.934897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.934849 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-home\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.934897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.934904 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bf6477-c859-4d11-8de8-aff72c7d849d-tls-certs\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.935157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.934965 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-model-cache\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.935157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.935038 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-tmp-dir\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.935157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.935092 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-dshm\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.935318 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.935162 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdjm\" (UniqueName: \"kubernetes.io/projected/06bf6477-c859-4d11-8de8-aff72c7d849d-kube-api-access-9hdjm\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.936985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.935802 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-tmp-dir\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.936985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.935868 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-home\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.936985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.935883 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-model-cache\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.937691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.937665 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-dshm\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.938479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.938460 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bf6477-c859-4d11-8de8-aff72c7d849d-tls-certs\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:47.943136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:47.943113 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdjm\" (UniqueName: \"kubernetes.io/projected/06bf6477-c859-4d11-8de8-aff72c7d849d-kube-api-access-9hdjm\") pod \"distinct-w3-39043d-kserve-54b776bfcc-llzcm\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:48.064416 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:48.064374 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:48.374034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:48.374001 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm"] Nov 03 19:44:48.374185 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:44:48.374082 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06bf6477_c859_4d11_8de8_aff72c7d849d.slice/crio-aaa43838153bd914aca6d73396e1084cd697399deb8062eb89f1f56688af5a1e WatchSource:0}: Error finding container aaa43838153bd914aca6d73396e1084cd697399deb8062eb89f1f56688af5a1e: Status 404 returned error can't find the container with id aaa43838153bd914aca6d73396e1084cd697399deb8062eb89f1f56688af5a1e Nov 03 19:44:49.086905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:49.086863 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" event={"ID":"06bf6477-c859-4d11-8de8-aff72c7d849d","Type":"ContainerStarted","Data":"548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2"} Nov 03 19:44:49.087348 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:49.086918 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" event={"ID":"06bf6477-c859-4d11-8de8-aff72c7d849d","Type":"ContainerStarted","Data":"aaa43838153bd914aca6d73396e1084cd697399deb8062eb89f1f56688af5a1e"} Nov 03 19:44:49.107082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:49.107026 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podStartSLOduration=2.107007347 podStartE2EDuration="2.107007347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:49.106798209 +0000 UTC m=+796.811637549" watchObservedRunningTime="2026-11-03 19:44:49.107007347 +0000 UTC m=+796.811846687" Nov 03 19:44:50.064783 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:50.064744 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:51.993703 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:44:51.993664 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:44:52.394679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:52.394555 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:52.407700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:52.407671 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:53.124367 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:53.124337 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:44:58.064919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:58.064869 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:58.083960 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:58.083928 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:44:58.160454 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:44:58.160418 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:45:00.171635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.171526 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq"] Nov 03 19:45:00.203307 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.203272 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq"] Nov 03 19:45:00.203663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.203641 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.249910 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.249884 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 19:45:00.306814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.306774 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.307019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.306963 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm44m\" (UniqueName: \"kubernetes.io/projected/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-kube-api-access-fm44m\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.408004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.407964 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.408196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.408109 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm44m\" (UniqueName: \"kubernetes.io/projected/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-kube-api-access-fm44m\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.408699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.408672 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.416303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.416269 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm44m\" (UniqueName: \"kubernetes.io/projected/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-kube-api-access-fm44m\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.521353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.521321 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:00.820045 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:00.820021 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq"] Nov 03 19:45:00.821938 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:00.821911 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488e1bc5_3be2_4ece_ad93_2ffdb3ee2582.slice/crio-e15c424292aa3665fb1c090dc994f4b43712fcfdab3dab1d6eb6289cbf08b079 WatchSource:0}: Error finding container e15c424292aa3665fb1c090dc994f4b43712fcfdab3dab1d6eb6289cbf08b079: Status 404 returned error can't find the container with id e15c424292aa3665fb1c090dc994f4b43712fcfdab3dab1d6eb6289cbf08b079 Nov 03 19:45:01.168517 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:01.168427 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" event={"ID":"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582","Type":"ContainerStarted","Data":"e15c424292aa3665fb1c090dc994f4b43712fcfdab3dab1d6eb6289cbf08b079"} Nov 03 19:45:02.491334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.491300 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth"] Nov 03 19:45:02.497006 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.496984 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.526130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.526103 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth"] Nov 03 19:45:02.526294 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.526167 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9"] Nov 03 19:45:02.536833 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.536799 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.536970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.536845 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbm2s\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-kube-api-access-rbm2s\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.537054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.536995 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.537054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.537037 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.537177 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.537090 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.537290 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.537269 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.538765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.538736 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.538926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.538776 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.538926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.538836 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.540958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.540938 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9"] Nov 03 19:45:02.541082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.541025 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.554688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.554460 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95"] Nov 03 19:45:02.559258 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.559239 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.581066 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.581039 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95"] Nov 03 19:45:02.639654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639608 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.639654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639654 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639680 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639697 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639735 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639756 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639789 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639808 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbm2s\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-kube-api-access-rbm2s\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639823 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639858 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.639877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639881 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639902 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639957 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a0624f-b804-49d4-9e59-d696bd226f1c-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.639976 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da839c7-b3f1-42b7-9fcd-709256732a3b-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640017 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640045 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640072 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640087 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsm4l\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-kube-api-access-gsm4l\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640125 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640140 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640161 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640186 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640207 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640224 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640252 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640284 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrkn\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-kube-api-access-fmrkn\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.640338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.640301 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.641158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.641105 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.641455 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.641432 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.641550 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.641448 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.641721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.641701 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.642188 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.642164 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.643056 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.643034 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.644494 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.644470 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.650499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.650479 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbm2s\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-kube-api-access-rbm2s\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.650774 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.650756 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.701252 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.701217 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:45:02.742279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742197 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrkn\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-kube-api-access-fmrkn\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742232 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.742279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742269 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742297 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742448 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742492 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.742731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742547 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.742731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742608 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742650 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742736 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742767 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a0624f-b804-49d4-9e59-d696bd226f1c-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742805 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da839c7-b3f1-42b7-9fcd-709256732a3b-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.742891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742845 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.742891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742888 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742945 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742975 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsm4l\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-kube-api-access-gsm4l\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742993 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.742995 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.743271 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.743313 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.743350 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.743527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.743380 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.743527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.743406 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.743776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.743753 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.744102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.743914 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.744102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.744027 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.745125 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.745076 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.745244 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.745221 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.745315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.745287 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da839c7-b3f1-42b7-9fcd-709256732a3b-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.745429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.745408 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.745627 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.745574 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a0624f-b804-49d4-9e59-d696bd226f1c-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.746071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.746047 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.754137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.754113 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.754282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.754264 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsm4l\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-kube-api-access-gsm4l\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.754547 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.754522 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.754763 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.754745 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrkn\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-kube-api-access-fmrkn\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.838024 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.837991 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:02.861145 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.861118 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:02.869847 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.869821 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf"] Nov 03 19:45:02.874503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.874484 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.880865 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.880838 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:02.897697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.894126 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf"] Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.945745 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.945858 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.946053 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.946110 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf7p\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-kube-api-access-zqf7p\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.946217 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.946264 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.946299 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.946338 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:02.947446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:02.946378 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.047473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.047430 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.047736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.047712 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.049159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.047838 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.049159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.047879 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.049159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.047913 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.049159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.047964 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.066829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.053369 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.087061 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.083754 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.087061 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.084026 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.087061 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.084094 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf7p\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-kube-api-access-zqf7p\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.088541 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.088518 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.088935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.088908 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.089263 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.089247 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.090565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.090513 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.091572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.091300 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.091572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.091516 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.104061 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.104029 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf7p\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-kube-api-access-zqf7p\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.104789 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.104739 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.173791 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:03.173559 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf55eb93_f7cf_45ce_aff9_b0af32b48eef.slice/crio-42d3a82ccf8f66ce6c93214d6782715dcfedcbe6c7eb06d03ffbc08c6b3a06ee WatchSource:0}: Error finding container 42d3a82ccf8f66ce6c93214d6782715dcfedcbe6c7eb06d03ffbc08c6b3a06ee: Status 404 returned error can't find the container with id 42d3a82ccf8f66ce6c93214d6782715dcfedcbe6c7eb06d03ffbc08c6b3a06ee Nov 03 19:45:03.177012 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.176970 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.177130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.177061 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.177130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.177105 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.177889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.177868 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth"] Nov 03 19:45:03.195940 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.193072 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" event={"ID":"bf55eb93-f7cf-45ce-aff9-b0af32b48eef","Type":"ContainerStarted","Data":"42d3a82ccf8f66ce6c93214d6782715dcfedcbe6c7eb06d03ffbc08c6b3a06ee"} Nov 03 19:45:03.199784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.199570 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9"] Nov 03 19:45:03.202197 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:03.202169 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da839c7_b3f1_42b7_9fcd_709256732a3b.slice/crio-cfc5f329d57f8ab2b9c302037fbb8e281a203fa35de5b3b0abe2c2042578ec9d WatchSource:0}: Error finding container cfc5f329d57f8ab2b9c302037fbb8e281a203fa35de5b3b0abe2c2042578ec9d: Status 404 returned error can't find the container with id cfc5f329d57f8ab2b9c302037fbb8e281a203fa35de5b3b0abe2c2042578ec9d Nov 03 19:45:03.206501 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.206461 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.206629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.206548 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.206698 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.206639 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.231879 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.231855 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:03.335663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.334967 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95"] Nov 03 19:45:03.338616 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:03.338543 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a0624f_b804_49d4_9e59_d696bd226f1c.slice/crio-b97970cb1cbf83d0f4caca04688f40f26c708ab3af32d394c52bc2dea5f79c9e WatchSource:0}: Error finding container b97970cb1cbf83d0f4caca04688f40f26c708ab3af32d394c52bc2dea5f79c9e: Status 404 returned error can't find the container with id b97970cb1cbf83d0f4caca04688f40f26c708ab3af32d394c52bc2dea5f79c9e Nov 03 19:45:03.342507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.341180 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.342507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.341320 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.342507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.341373 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.571157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.571055 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf"] Nov 03 19:45:03.576867 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:03.576822 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb5bc4d_59ea_4f08_ba3c_459eea50fe46.slice/crio-b79261f3051e02560061591029ce38f6e96322bdd048f8195d4b6ac5301eef32 WatchSource:0}: Error finding container b79261f3051e02560061591029ce38f6e96322bdd048f8195d4b6ac5301eef32: Status 404 returned error can't find the container with id b79261f3051e02560061591029ce38f6e96322bdd048f8195d4b6ac5301eef32 Nov 03 19:45:03.579911 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.579875 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.580002 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.579964 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:03.580064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:03.580010 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:04.278802 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.278763 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" event={"ID":"bf55eb93-f7cf-45ce-aff9-b0af32b48eef","Type":"ContainerStarted","Data":"7842ce2dcee25666ac21904b78eca3b756d16fcd29413757b415a7ad9f5d31ed"} Nov 03 19:45:04.331262 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.331199 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" event={"ID":"a4a0624f-b804-49d4-9e59-d696bd226f1c","Type":"ContainerStarted","Data":"b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8"} Nov 03 19:45:04.343605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.336565 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" event={"ID":"a4a0624f-b804-49d4-9e59-d696bd226f1c","Type":"ContainerStarted","Data":"b97970cb1cbf83d0f4caca04688f40f26c708ab3af32d394c52bc2dea5f79c9e"} Nov 03 19:45:04.382742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.382696 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" event={"ID":"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46","Type":"ContainerStarted","Data":"7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f"} Nov 03 19:45:04.383023 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.383002 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" event={"ID":"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46","Type":"ContainerStarted","Data":"b79261f3051e02560061591029ce38f6e96322bdd048f8195d4b6ac5301eef32"} Nov 03 19:45:04.401788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.400856 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" event={"ID":"2da839c7-b3f1-42b7-9fcd-709256732a3b","Type":"ContainerStarted","Data":"cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6"} Nov 03 19:45:04.401788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.400926 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" event={"ID":"2da839c7-b3f1-42b7-9fcd-709256732a3b","Type":"ContainerStarted","Data":"cfc5f329d57f8ab2b9c302037fbb8e281a203fa35de5b3b0abe2c2042578ec9d"} Nov 03 19:45:04.469604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.463264 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podStartSLOduration=2.463244027 podStartE2EDuration="2.463244027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.427436908 +0000 UTC m=+812.132276277" watchObservedRunningTime="2026-11-03 19:45:04.463244027 +0000 UTC m=+812.168083368" Nov 03 19:45:04.641665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.625886 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podStartSLOduration=2.625864994 podStartE2EDuration="2.625864994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.461189625 +0000 UTC m=+812.166028966" watchObservedRunningTime="2026-11-03 19:45:04.625864994 +0000 UTC m=+812.330704336" Nov 03 19:45:04.660338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.660265 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podStartSLOduration=2.660229925 podStartE2EDuration="2.660229925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.518019237 +0000 UTC m=+812.222858592" watchObservedRunningTime="2026-11-03 19:45:04.660229925 +0000 UTC m=+812.365069266" Nov 03 19:45:04.694603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.691447 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podStartSLOduration=2.691423673 podStartE2EDuration="2.691423673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:04.598184284 +0000 UTC m=+812.303023625" watchObservedRunningTime="2026-11-03 19:45:04.691423673 +0000 UTC m=+812.396263014" Nov 03 19:45:04.860701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.860654 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:04.864710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.864665 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:04.864938 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.864912 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:04.890935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.890907 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:04.891218 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.891190 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:04.920188 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.920089 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:04.920490 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.920456 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:04.920748 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.920731 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:04.920900 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:04.920880 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:05.148658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.140673 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.177656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.173878 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.190604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.186704 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.247604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.245722 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:05.247604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.245871 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:05.247604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.245914 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:05.324621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.318680 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.637747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.637683 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.739823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.739760 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.801729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.801664 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.844614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.844032 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:05.844614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.844091 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:05.871432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.870175 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:05.873489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.872392 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:05.873489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.872437 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:05.882241 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.882215 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:05.882456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:05.882433 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:06.147736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.147679 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:06.152342 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.152314 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:45:06.190893 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.190852 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:06.191741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.191718 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:45:06.193459 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.193429 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:06.193929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.193911 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:45:06.563685 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.563644 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:06.563973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.563940 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:06.787009 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.786955 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:06.866666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.850316 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:45:06.960607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.923851 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:07.011604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.973011 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:07.011604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:06.950851 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:07.011604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.010894 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:07.011604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.010956 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:07.142657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.142498 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:07.151752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.151707 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:07.226721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.185100 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:07.226721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.185163 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:07.227792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.227501 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:07.228292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.228264 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:45:07.616099 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.616061 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:07.675717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.675670 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:07.774654 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:07.770181 4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.258s" Nov 03 19:45:07.774654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.770322 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:45:07.774654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.770361 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:45:07.774654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.770395 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:45:07.774654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.770427 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:45:07.793692 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.793651 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:07.860728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.860679 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:07.983770 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.983677 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:08.005605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.983980 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:08.005890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.996030 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:08.006017 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.005993 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:08.006132 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:07.999660 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:08.006231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.006215 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:08.033606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.016826 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:08.155610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.151332 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:08.188404 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.168197 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:08.265641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.255740 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:08.265641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.255802 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:08.265641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.255867 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:45:08.351606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.339935 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:08.874606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.873764 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:08.874606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.873829 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:08.886780 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.886749 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:08.887030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.886995 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:08.912855 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.912825 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:08.913078 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.913053 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:08.967317 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.967278 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:08.983673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:08.975694 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:09.132811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.132699 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:09.170612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.133349 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:09.190154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.159918 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:09.253296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.227130 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:45:09.253877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.235070 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:09.254084 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.254056 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:09.256543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.256505 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:09.340614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.340112 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:09.392668 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.387885 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:45:09.854803 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.854758 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:09.855119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.855071 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:09.873613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.867593 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:09.873868 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.873841 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:09.931776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.931739 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:09.932346 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:09.932312 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:10.083414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.083357 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:10.084073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.084039 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:10.221087 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.220986 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:10.513206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.302764 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:10.513498 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.513460 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:10.513659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.309852 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:10.514524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.309547 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:10.517920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.342533 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:10.840677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.840145 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:10.870923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.870830 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:10.921638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.917122 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:10.921638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:10.917187 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:11.014308 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.008794 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:11.070409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.070370 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:11.096897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.089724 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:11.096897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.090102 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:45:11.206727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.206679 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:45:11.238288 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.238250 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:11.240462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.240428 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:11.287631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.274998 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:45:11.348811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.348708 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:11.408865 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.408811 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:11.641702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.618227 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:11.641702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.618608 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:11.641702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.618885 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:11.927499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.909527 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:11.927499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.909607 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:11.927499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.911600 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:11.927499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.911644 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:11.927499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.911713 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:11.927499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:11.911731 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:12.121608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.095808 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:12.121608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.096201 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:12.211113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.208427 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:12.211113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.209135 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:45:12.251121 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.251086 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:12.251380 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.251350 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:12.667769 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.667712 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:12.698700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.670666 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:12.724736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.670749 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:12.871608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.863836 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:12.871608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.863897 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:12.871608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.864207 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:12.871608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.864251 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:12.895805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.895752 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:12.896044 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:12.896019 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:13.108692 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.108646 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:13.241126 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.108955 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:45:13.242324 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.218734 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:13.242761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.240804 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:13.257716 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.257676 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:13.482671 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.453113 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:13.701725 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.701665 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:13.732183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.731088 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:13.744500 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.744463 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:13.837897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.837864 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:45:13.838524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.838492 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:45:13.838661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.838549 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:45:13.891608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.886820 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:13.891608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.886880 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:13.891608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.886889 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:13.891608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:13.887030 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:14.130482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.130398 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:45:14.184669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.184541 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:45:14.196137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.194927 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:45:14.236814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.236783 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 19:45:14.237082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.237056 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 19:45:14.241075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.241033 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:14.254646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.244642 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:14.323603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.322442 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:45:14.367789 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.362446 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:45:14.863296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.862836 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:45:14.863296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.862898 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:45:14.883609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.882787 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 19:45:14.883609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:14.882843 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 19:45:15.617604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.617398 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:15.621602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.618792 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:15.626609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.624422 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 19:45:15.821881 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.821687 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:15.822843 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.822727 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:15.828358 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.828335 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:45:15.945652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.945422 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:15.951640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.945828 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:15.951640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.946693 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 19:45:15.978614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.974852 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:15.978614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.975689 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:15.978614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:15.976767 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:45:18.445297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:18.445261 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" event={"ID":"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582","Type":"ContainerStarted","Data":"3e12dd1fc71bd6b8e40c28551246c9d7c2092e2526af08ce4224d9e4cfb95355"} Nov 03 19:45:18.461473 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:18.461417 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" podStartSLOduration=1.780587674 podStartE2EDuration="18.461401135s" totalImagesPullingTime="16.680813461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:18.459004003 +0000 UTC m=+826.163843344" watchObservedRunningTime="2026-11-03 19:45:18.461401135 +0000 UTC m=+826.166240474" Nov 03 19:45:19.453918 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:19.453876 4314 generic.go:356] "Generic (PLEG): container finished" podID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerID="3e12dd1fc71bd6b8e40c28551246c9d7c2092e2526af08ce4224d9e4cfb95355" exitCode=60 Nov 03 19:45:19.454400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:19.453942 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" event={"ID":"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582","Type":"ContainerDied","Data":"3e12dd1fc71bd6b8e40c28551246c9d7c2092e2526af08ce4224d9e4cfb95355"} Nov 03 19:45:19.454400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:19.454330 4314 scope.go:122] "RemoveContainer" containerID="3e12dd1fc71bd6b8e40c28551246c9d7c2092e2526af08ce4224d9e4cfb95355" Nov 03 19:45:20.462118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:20.462077 4314 generic.go:356] "Generic (PLEG): container finished" podID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerID="2062b89dee9c7c7fcc517386fc3a05cb5e7598cc184c05736be88fb5aeda49f1" exitCode=60 Nov 03 19:45:20.462534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:20.462161 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" event={"ID":"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582","Type":"ContainerDied","Data":"2062b89dee9c7c7fcc517386fc3a05cb5e7598cc184c05736be88fb5aeda49f1"} Nov 03 19:45:20.462534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:20.462222 4314 scope.go:122] "RemoveContainer" containerID="3e12dd1fc71bd6b8e40c28551246c9d7c2092e2526af08ce4224d9e4cfb95355" Nov 03 19:45:20.462534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:20.462482 4314 scope.go:122] "RemoveContainer" containerID="2062b89dee9c7c7fcc517386fc3a05cb5e7598cc184c05736be88fb5aeda49f1" Nov 03 19:45:20.462851 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:20.462830 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq_odh-ai-gateway-infra(488e1bc5-3be2-4ece-ad93-2ffdb3ee2582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" Nov 03 19:45:21.473667 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:21.473634 4314 scope.go:122] "RemoveContainer" containerID="2062b89dee9c7c7fcc517386fc3a05cb5e7598cc184c05736be88fb5aeda49f1" Nov 03 19:45:21.474046 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:21.473940 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq_odh-ai-gateway-infra(488e1bc5-3be2-4ece-ad93-2ffdb3ee2582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" Nov 03 19:45:21.998033 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:21.997992 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:45:25.791607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:25.786414 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp"] Nov 03 19:45:26.039659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:25.992033 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp"] Nov 03 19:45:26.039659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:25.992142 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.039659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.014042 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39043d\"/\"distinct-2-w3-39043d-kserve-self-signed-certs\"" Nov 03 19:45:26.055644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.045103 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-tmp-dir\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.055644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.045163 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad274c2-4168-4b70-a353-52475635630d-tls-certs\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.055644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.045212 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-home\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.055644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.045289 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-dshm\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.055644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.045312 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn6c8\" (UniqueName: \"kubernetes.io/projected/aad274c2-4168-4b70-a353-52475635630d-kube-api-access-vn6c8\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.055644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.045395 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-model-cache\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.152606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.148417 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad274c2-4168-4b70-a353-52475635630d-tls-certs\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.152606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.148512 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-home\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.152606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.148627 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-dshm\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.152606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.148654 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn6c8\" (UniqueName: \"kubernetes.io/projected/aad274c2-4168-4b70-a353-52475635630d-kube-api-access-vn6c8\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.152606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.148758 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-model-cache\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.152606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.148885 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-tmp-dir\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.152606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.149459 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-tmp-dir\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.174605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.171792 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-home\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.273512 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.193502 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:26.274811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.273703 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:26.317273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.312656 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:45:26.317273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.313857 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:26.317273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.315990 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:26.328718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.328692 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-model-cache\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.342728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.342693 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:26.406602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.392882 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-dshm\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.406602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.400881 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad274c2-4168-4b70-a353-52475635630d-tls-certs\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.484878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.469313 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn6c8\" (UniqueName: \"kubernetes.io/projected/aad274c2-4168-4b70-a353-52475635630d-kube-api-access-vn6c8\") pod \"distinct-2-w3-39043d-kserve-75c6595994-frbsp\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:26.768657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:26.756863 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:27.072658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.046573 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:45:27.218165 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.218122 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:27.274060 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.274004 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:27.355250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.355152 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:27.637665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.355522 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:27.638283 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.359455 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:27.638818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.359487 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:45:27.639232 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:27.433954 4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.005s" Nov 03 19:45:27.639554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.639534 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:45:27.641653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.641634 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:45:27.642804 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.642786 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:45:27.642990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.642975 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:45:27.662608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.648685 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:45:27.662959 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.662936 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:45:27.848659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.847614 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:45:27.895664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.894674 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:45:27.895664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.894742 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:27.925734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:27.920090 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:45:28.080318 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.080291 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:45:28.080889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.076727 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:28.086913 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.086884 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:45:28.104505 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.104471 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:45:28.107413 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.107381 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:45:28.121543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.120542 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:45:28.237694 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.237662 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:45:28.487092 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.487061 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp"] Nov 03 19:45:28.489366 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:28.489338 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad274c2_4168_4b70_a353_52475635630d.slice/crio-a7e3308848f2953809416e51fe5c5fe4b5e8c9b239e915e9ee131481f0e7b8de WatchSource:0}: Error finding container a7e3308848f2953809416e51fe5c5fe4b5e8c9b239e915e9ee131481f0e7b8de: Status 404 returned error can't find the container with id a7e3308848f2953809416e51fe5c5fe4b5e8c9b239e915e9ee131481f0e7b8de Nov 03 19:45:28.521930 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:28.521899 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" event={"ID":"aad274c2-4168-4b70-a353-52475635630d","Type":"ContainerStarted","Data":"a7e3308848f2953809416e51fe5c5fe4b5e8c9b239e915e9ee131481f0e7b8de"} Nov 03 19:45:29.529779 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:29.529731 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" event={"ID":"aad274c2-4168-4b70-a353-52475635630d","Type":"ContainerStarted","Data":"0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71"} Nov 03 19:45:29.547534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:29.547484 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podStartSLOduration=4.547470702 podStartE2EDuration="4.547470702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:29.545661061 +0000 UTC m=+837.250500401" watchObservedRunningTime="2026-11-03 19:45:29.547470702 +0000 UTC m=+837.252310043" Nov 03 19:45:29.772203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:29.772173 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:32.648599 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.648548 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8"] Nov 03 19:45:32.693308 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.693278 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8"] Nov 03 19:45:32.693641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.693575 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790261 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790318 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfckw\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-kube-api-access-sfckw\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790407 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790433 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790468 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790500 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790660 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1abade-07f5-49a0-8fa3-4685887748e2-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790728 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.790852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.790791 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901539 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901427 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901539 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901493 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901604 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901666 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfckw\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-kube-api-access-sfckw\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901703 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901726 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901774 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.901800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901800 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.902100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901950 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1abade-07f5-49a0-8fa3-4685887748e2-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.902100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.901955 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.902100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.902012 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.902538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.902276 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.902753 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.902683 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.903522 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.903495 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1abade-07f5-49a0-8fa3-4685887748e2-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.906304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.906277 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.908441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.908415 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.911137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.911073 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:32.911338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:32.911316 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfckw\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-kube-api-access-sfckw\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:33.026441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.026411 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:33.035161 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.035133 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts"] Nov 03 19:45:33.040098 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.040080 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.049015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.048992 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts"] Nov 03 19:45:33.104232 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.104197 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.104643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.104461 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5r9q\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-kube-api-access-t5r9q\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.104828 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.104809 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.104984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.104968 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.105187 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.105169 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.105349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.105333 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.105575 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.105559 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.105823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.105803 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.105941 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.105928 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.207997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.207907 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.207997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.207947 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5r9q\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-kube-api-access-t5r9q\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.207997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.207984 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.208281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.208012 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.208497 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.208068 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.208596 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.208537 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.208596 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.208351 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.208704 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.208481 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.208849 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.208827 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.209343 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.209032 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.209427 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.209380 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.209488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.209429 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.209488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.209324 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.209488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.209248 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.210931 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.210905 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.211597 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.211560 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.217711 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.217688 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.218012 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.217990 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5r9q\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-kube-api-access-t5r9q\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.351141 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.351115 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8"] Nov 03 19:45:33.352310 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:33.352278 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1abade_07f5_49a0_8fa3_4685887748e2.slice/crio-0bd4c319cf1b7b04a4bec061983fef5ff46333b9b2d3dcb548f0dedb0311558f WatchSource:0}: Error finding container 0bd4c319cf1b7b04a4bec061983fef5ff46333b9b2d3dcb548f0dedb0311558f: Status 404 returned error can't find the container with id 0bd4c319cf1b7b04a4bec061983fef5ff46333b9b2d3dcb548f0dedb0311558f Nov 03 19:45:33.354999 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.354966 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:33.355072 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.355030 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:33.355072 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.355060 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:33.390488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.390469 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:33.562855 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.561574 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" event={"ID":"5d1abade-07f5-49a0-8fa3-4685887748e2","Type":"ContainerStarted","Data":"3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8"} Nov 03 19:45:33.562855 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.561660 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" event={"ID":"5d1abade-07f5-49a0-8fa3-4685887748e2","Type":"ContainerStarted","Data":"0bd4c319cf1b7b04a4bec061983fef5ff46333b9b2d3dcb548f0dedb0311558f"} Nov 03 19:45:33.591826 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.591767 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podStartSLOduration=1.591747215 podStartE2EDuration="1.591747215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:33.590552712 +0000 UTC m=+841.295392052" watchObservedRunningTime="2026-11-03 19:45:33.591747215 +0000 UTC m=+841.296586559" Nov 03 19:45:33.681754 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.681719 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts"] Nov 03 19:45:33.684484 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:33.684458 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9892a5d_a52f_4f6a_b5ae_f8d926cb283a.slice/crio-e7c608eaaf8abdc8cff9ee71b2d720bb74db6e7602f8b7d39eb8c215cd19ed1c WatchSource:0}: Error finding container e7c608eaaf8abdc8cff9ee71b2d720bb74db6e7602f8b7d39eb8c215cd19ed1c: Status 404 returned error can't find the container with id e7c608eaaf8abdc8cff9ee71b2d720bb74db6e7602f8b7d39eb8c215cd19ed1c Nov 03 19:45:33.688080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.688038 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:33.688177 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.688133 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:33.688234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:33.688182 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:34.027272 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:34.027238 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:34.029248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:34.029214 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:45:34.029378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:34.029268 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:45:34.424429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:34.424341 4314 scope.go:122] "RemoveContainer" containerID="2062b89dee9c7c7fcc517386fc3a05cb5e7598cc184c05736be88fb5aeda49f1" Nov 03 19:45:34.620608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:34.611815 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" event={"ID":"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a","Type":"ContainerStarted","Data":"3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301"} Nov 03 19:45:34.620608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:34.611877 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" event={"ID":"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a","Type":"ContainerStarted","Data":"e7c608eaaf8abdc8cff9ee71b2d720bb74db6e7602f8b7d39eb8c215cd19ed1c"} Nov 03 19:45:34.640970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:34.640919 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podStartSLOduration=1.640901889 podStartE2EDuration="1.640901889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:34.636796497 +0000 UTC m=+842.341635838" watchObservedRunningTime="2026-11-03 19:45:34.640901889 +0000 UTC m=+842.345741228" Nov 03 19:45:35.026996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:35.026964 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:45:35.027336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:35.027024 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:45:35.401634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:35.400801 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:35.401634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:35.400927 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:45:35.401634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:35.401004 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:45:35.629097 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:35.629062 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" event={"ID":"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582","Type":"ContainerStarted","Data":"20a7c8a860996ad8f09388af6d898827bff298e1486af418c2f8fbbed863a0f0"} Nov 03 19:45:36.026803 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.026750 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:45:36.027012 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.026843 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:45:36.393661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.392961 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:45:36.393661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.393021 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:45:36.639209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.639174 4314 generic.go:356] "Generic (PLEG): container finished" podID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerID="20a7c8a860996ad8f09388af6d898827bff298e1486af418c2f8fbbed863a0f0" exitCode=60 Nov 03 19:45:36.640020 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.639996 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" event={"ID":"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582","Type":"ContainerDied","Data":"20a7c8a860996ad8f09388af6d898827bff298e1486af418c2f8fbbed863a0f0"} Nov 03 19:45:36.640073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.640051 4314 scope.go:122] "RemoveContainer" containerID="2062b89dee9c7c7fcc517386fc3a05cb5e7598cc184c05736be88fb5aeda49f1" Nov 03 19:45:36.640418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.640402 4314 scope.go:122] "RemoveContainer" containerID="20a7c8a860996ad8f09388af6d898827bff298e1486af418c2f8fbbed863a0f0" Nov 03 19:45:36.640850 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:36.640832 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq_odh-ai-gateway-infra(488e1bc5-3be2-4ece-ad93-2ffdb3ee2582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" Nov 03 19:45:36.685962 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.685854 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq"] Nov 03 19:45:36.788174 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.788138 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:36.848992 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:36.848961 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:37.027494 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:37.027457 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:45:37.028016 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:37.027510 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:45:37.398104 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:37.397993 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:45:37.398357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:37.398327 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:45:37.738774 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:37.738546 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:38.027736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.027175 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:45:38.027736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.027236 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:45:38.240327 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.240295 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:38.335659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.335052 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-maas-api-service-ca\") pod \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " Nov 03 19:45:38.335659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.335116 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-kube-api-access-fm44m\" (UniqueName: \"kubernetes.io/projected/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-kube-api-access-fm44m\") pod \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\" (UID: \"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582\") " Nov 03 19:45:38.338270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.337545 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-maas-api-service-ca" pod "488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" (UID: "488e1bc5-3be2-4ece-ad93-2ffdb3ee2582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:45:38.354690 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.351726 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-kube-api-access-fm44m" pod "488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" (UID: "488e1bc5-3be2-4ece-ad93-2ffdb3ee2582"). InnerVolumeSpecName "kube-api-access-fm44m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:45:38.409158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.396977 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:45:38.409158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.397044 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:45:38.438813 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.438776 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:38.438813 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.438814 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm44m\" (UniqueName: \"kubernetes.io/projected/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582-kube-api-access-fm44m\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:45:38.697800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.697695 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" event={"ID":"488e1bc5-3be2-4ece-ad93-2ffdb3ee2582","Type":"ContainerDied","Data":"e15c424292aa3665fb1c090dc994f4b43712fcfdab3dab1d6eb6289cbf08b079"} Nov 03 19:45:38.698075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.698057 4314 scope.go:122] "RemoveContainer" containerID="20a7c8a860996ad8f09388af6d898827bff298e1486af418c2f8fbbed863a0f0" Nov 03 19:45:38.698220 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.697888 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq" Nov 03 19:45:38.800162 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.800129 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq"] Nov 03 19:45:38.827352 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:38.818171 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895585-dgdrq"] Nov 03 19:45:39.027395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:39.027347 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:45:39.027677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:39.027637 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:45:39.391075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:39.390981 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:45:39.392459 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:39.392417 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:45:40.027618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:40.027564 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:45:40.027796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:40.027641 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:45:40.397708 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:40.397611 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:45:40.398334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:40.398292 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:45:40.432309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:40.432276 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" path="/var/lib/kubelet/pods/488e1bc5-3be2-4ece-ad93-2ffdb3ee2582/volumes" Nov 03 19:45:41.107786 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:41.106858 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:41.109019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:41.108870 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:41.126441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:41.121410 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 19:45:41.409739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:41.409645 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:41.410219 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:41.410063 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:41.410956 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:41.410934 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 19:45:47.657490 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.657404 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d"] Nov 03 19:45:47.658075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.658057 4314 cpu_manager.go:414] "containerMap: removing container" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.658075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.658074 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.658207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.658087 4314 cpu_manager.go:414] "containerMap: removing container" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.658207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.658092 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.658306 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.658217 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.658306 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.658230 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.658306 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.658242 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.663056 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.663033 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.673388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.672941 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d"] Nov 03 19:45:47.723537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.723502 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl"] Nov 03 19:45:47.724396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.724376 4314 cpu_manager.go:414] "containerMap: removing container" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.724474 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.724397 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488e1bc5-3be2-4ece-ad93-2ffdb3ee2582" containerName="cleanup" Nov 03 19:45:47.727114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727089 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727158 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879tb\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-kube-api-access-879tb\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727261 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727241 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727294 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727425 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727505 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727477 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727621 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5607bf06-0901-4fcc-8914-5cd2e90212d8-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727678 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.727762 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.727745 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.728849 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.728754 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.737407 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.737349 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl"] Nov 03 19:45:47.747875 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.743248 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm"] Nov 03 19:45:47.769623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.766947 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm"] Nov 03 19:45:47.769623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.767150 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.829532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.829498 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.829706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.829561 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.829751 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.829693 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.829810 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.829779 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.829857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.829821 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmvxv\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-kube-api-access-hmvxv\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.829857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.829838 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efee946b-04fd-4e86-b1dd-8190c606fb4a-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.829993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.829976 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xc4\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-kube-api-access-r9xc4\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830002 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830015 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879tb\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-kube-api-access-879tb\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830134 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830066 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830134 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830122 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830163 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.830209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830182 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830199 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.830331 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830283 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830331 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830324 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830351 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830379 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830390 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5607bf06-0901-4fcc-8914-5cd2e90212d8-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830413 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830439 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830485 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830523 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.830765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830612 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.830765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830687 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830727 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.830876 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830757 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.830876 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830779 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830876 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830862 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830883 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2732e9-41b4-4072-a88f-8c02ab3863da-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.830988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.830962 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.831065 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.831048 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5607bf06-0901-4fcc-8914-5cd2e90212d8-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.832806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.832786 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.833045 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.833019 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.837570 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.837548 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.837936 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.837917 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-879tb\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-kube-api-access-879tb\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:47.927057 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.926973 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9"] Nov 03 19:45:47.932032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932004 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.932213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932050 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.932213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932078 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932346 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932300 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932471 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932412 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932462 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.932527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932520 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.932623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932544 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2732e9-41b4-4072-a88f-8c02ab3863da-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.932623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932552 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932644 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.932747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932704 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932740 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932769 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmvxv\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-kube-api-access-hmvxv\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932791 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efee946b-04fd-4e86-b1dd-8190c606fb4a-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.932934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932862 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xc4\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-kube-api-access-r9xc4\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.932934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.932912 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.933019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.933002 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.933064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.933041 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.933102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.933061 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.933102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.933072 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.933102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.933080 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.933387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.933356 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.934136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.934111 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efee946b-04fd-4e86-b1dd-8190c606fb4a-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.934943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.934896 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.935184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.935154 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:47.935277 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.935205 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2732e9-41b4-4072-a88f-8c02ab3863da-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.935394 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.935371 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.938620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.938284 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.938620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.938517 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.938975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.938954 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.941525 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.941131 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.941525 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.941484 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.942328 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.942305 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9"] Nov 03 19:45:47.942853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.942831 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.946029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.946006 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:47.948347 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.948323 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.949994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.948656 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xc4\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-kube-api-access-r9xc4\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:47.950742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:47.950719 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmvxv\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-kube-api-access-hmvxv\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:48.004079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.003870 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:48.034687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.034647 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.034841 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.034694 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93af09a3-1a47-43b8-982b-33a3df9d5283-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.034841 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.034802 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.034981 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.034962 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.035026 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.035008 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5l4\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-kube-api-access-sd5l4\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.035127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.035103 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.035183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.035172 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.035232 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.035215 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.035310 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.035292 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.070293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.069501 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:48.087254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.085942 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136347 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136436 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136524 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136572 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93af09a3-1a47-43b8-982b-33a3df9d5283-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136722 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136887 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136943 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5l4\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-kube-api-access-sd5l4\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.136998 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137356 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.137046 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.137496 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.137921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.137537 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.138535 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.138514 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93af09a3-1a47-43b8-982b-33a3df9d5283-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.139033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.138762 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.141573 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.141539 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.144063 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.144039 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.146309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.146269 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.152575 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.152541 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.155365 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.155313 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5l4\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-kube-api-access-sd5l4\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.214249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.208961 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s"] Nov 03 19:45:48.226687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.226431 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79"] Nov 03 19:45:48.228032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.226574 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.235292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.233705 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s"] Nov 03 19:45:48.235292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.233777 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79"] Nov 03 19:45:48.235292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.233895 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.310835 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.308002 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:48.316697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.314206 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.316697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.314303 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.316697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.314354 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.317091 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.316836 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d"] Nov 03 19:45:48.343749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343564 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54734b03-0f73-4c72-a11f-9af1c765b89b-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.343749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343677 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.343749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343720 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.343920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343750 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.343920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343778 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.343920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343860 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.344079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343921 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mpl\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-kube-api-access-m8mpl\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.344079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.343960 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.344079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344014 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.344079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344069 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.344275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344103 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.344275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344128 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.344275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344176 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.344275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344254 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.344469 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344287 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4mk\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-kube-api-access-9l4mk\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.344469 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344349 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.344469 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344430 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.344631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.344480 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445455 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445512 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445538 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445563 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445649 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445696 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mpl\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-kube-api-access-m8mpl\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445733 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445789 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445835 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445868 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445890 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.445996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.445937 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.446739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.446010 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.446739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.446042 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4mk\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-kube-api-access-9l4mk\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.446739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.446101 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.446739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.446181 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.446739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.446226 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.446739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.446327 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54734b03-0f73-4c72-a11f-9af1c765b89b-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.447247 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.447353 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54734b03-0f73-4c72-a11f-9af1c765b89b-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.447367 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.447667 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.447762 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.447918 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.448181 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.448311 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.448364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.448328 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.448930 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.448549 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.451789 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.451698 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.452641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.452180 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.453411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.453354 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.454096 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.454070 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.455299 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.455275 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4mk\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-kube-api-access-9l4mk\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.456061 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.456021 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.457550 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.457489 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.459681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.459653 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mpl\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-kube-api-access-m8mpl\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.621153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.621116 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:48.630227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.623083 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9"] Nov 03 19:45:48.633698 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.633670 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:48.634371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.634351 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm"] Nov 03 19:45:48.637862 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.637839 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl"] Nov 03 19:45:48.639448 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:48.639417 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2732e9_41b4_4072_a88f_8c02ab3863da.slice/crio-960b21d960846f538bfbde0c6af55da6c3f897e06cb38bee9b546181908d6597 WatchSource:0}: Error finding container 960b21d960846f538bfbde0c6af55da6c3f897e06cb38bee9b546181908d6597: Status 404 returned error can't find the container with id 960b21d960846f538bfbde0c6af55da6c3f897e06cb38bee9b546181908d6597 Nov 03 19:45:48.640568 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:48.640547 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93af09a3_1a47_43b8_982b_33a3df9d5283.slice/crio-842de7725d3633a90d449bbdf06b05fb13bb43729cadb78d6f96eca22b75859b WatchSource:0}: Error finding container 842de7725d3633a90d449bbdf06b05fb13bb43729cadb78d6f96eca22b75859b: Status 404 returned error can't find the container with id 842de7725d3633a90d449bbdf06b05fb13bb43729cadb78d6f96eca22b75859b Nov 03 19:45:48.641654 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:45:48.641634 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefee946b_04fd_4e86_b1dd_8190c606fb4a.slice/crio-e192a7b6d969a5d2ffd1f847b05269d3f4dce7014a78c3a62e5726846835db77 WatchSource:0}: Error finding container e192a7b6d969a5d2ffd1f847b05269d3f4dce7014a78c3a62e5726846835db77: Status 404 returned error can't find the container with id e192a7b6d969a5d2ffd1f847b05269d3f4dce7014a78c3a62e5726846835db77 Nov 03 19:45:48.642654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.642622 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.642761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.642693 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.642761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.642739 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.649419 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.649384 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.649502 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.649456 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.649613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.649499 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.649613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.649443 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.649724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.649640 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.649724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.649690 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:48.814642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.814564 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" event={"ID":"5607bf06-0901-4fcc-8914-5cd2e90212d8","Type":"ContainerStarted","Data":"489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a"} Nov 03 19:45:48.815064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.815040 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" event={"ID":"5607bf06-0901-4fcc-8914-5cd2e90212d8","Type":"ContainerStarted","Data":"dfacbf6782bd71e6bb0e091919e8cf028584f7e568c48b5770a4ca8074a2bf9d"} Nov 03 19:45:48.816809 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.816783 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" event={"ID":"4a2732e9-41b4-4072-a88f-8c02ab3863da","Type":"ContainerStarted","Data":"960b21d960846f538bfbde0c6af55da6c3f897e06cb38bee9b546181908d6597"} Nov 03 19:45:48.818541 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.818518 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" event={"ID":"efee946b-04fd-4e86-b1dd-8190c606fb4a","Type":"ContainerStarted","Data":"e192a7b6d969a5d2ffd1f847b05269d3f4dce7014a78c3a62e5726846835db77"} Nov 03 19:45:48.819617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.819562 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" event={"ID":"93af09a3-1a47-43b8-982b-33a3df9d5283","Type":"ContainerStarted","Data":"842de7725d3633a90d449bbdf06b05fb13bb43729cadb78d6f96eca22b75859b"} Nov 03 19:45:48.893617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:48.875428 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podStartSLOduration=1.8754035660000001 podStartE2EDuration="1.875403566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:48.836467801 +0000 UTC m=+856.541307145" watchObservedRunningTime="2026-11-03 19:45:48.875403566 +0000 UTC m=+856.580242931" Nov 03 19:45:49.014608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.014465 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:45:49.016360 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.015752 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:49.016360 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.015809 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:49.041313 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.040329 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79"] Nov 03 19:45:49.042483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.042441 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:49.042571 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.042529 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:49.042655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.042571 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:49.045414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.045155 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s"] Nov 03 19:45:49.047860 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.046009 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:49.047860 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.046088 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:49.047860 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.046131 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:45:49.890275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.890232 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" event={"ID":"efee946b-04fd-4e86-b1dd-8190c606fb4a","Type":"ContainerStarted","Data":"0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613"} Nov 03 19:45:49.904790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.904748 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" event={"ID":"93af09a3-1a47-43b8-982b-33a3df9d5283","Type":"ContainerStarted","Data":"6a5dd594d30c853dece4ad75554120f83c7e57e52f85202ed1c2aa7adde2823a"} Nov 03 19:45:49.935633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.935592 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" event={"ID":"54734b03-0f73-4c72-a11f-9af1c765b89b","Type":"ContainerStarted","Data":"ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0"} Nov 03 19:45:49.935908 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.935882 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" event={"ID":"54734b03-0f73-4c72-a11f-9af1c765b89b","Type":"ContainerStarted","Data":"763c8acf8598b5f3f8f8727116ab05625908725a4d1f0f758c40750e9b9c7b49"} Nov 03 19:45:50.016759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.958239 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podStartSLOduration=2.9582150350000003 podStartE2EDuration="2.958215035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.916088321 +0000 UTC m=+857.620927685" watchObservedRunningTime="2026-11-03 19:45:49.958215035 +0000 UTC m=+857.663054461" Nov 03 19:45:50.019636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:49.961091 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" event={"ID":"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a","Type":"ContainerStarted","Data":"f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e"} Nov 03 19:45:50.019890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.019861 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" event={"ID":"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a","Type":"ContainerStarted","Data":"fe4e0e349c8072473d4b42e5f26479c5c6e041fd757500ed87d4f6b572856098"} Nov 03 19:45:50.020049 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.020033 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" event={"ID":"4a2732e9-41b4-4072-a88f-8c02ab3863da","Type":"ContainerStarted","Data":"19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b"} Nov 03 19:45:50.023676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.020478 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podStartSLOduration=3.020465953 podStartE2EDuration="3.020465953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.934918887 +0000 UTC m=+857.639758227" watchObservedRunningTime="2026-11-03 19:45:50.020465953 +0000 UTC m=+857.725305291" Nov 03 19:45:50.112133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.112098 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:45:50.134937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.134905 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:45:50.135199 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.127287 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podStartSLOduration=2.1272682290000002 podStartE2EDuration="2.127268229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:50.02761504 +0000 UTC m=+857.732454370" watchObservedRunningTime="2026-11-03 19:45:50.127268229 +0000 UTC m=+857.832107573" Nov 03 19:45:50.138719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.126931 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:50.138920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.138892 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:50.155941 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.138682 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:50.156188 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.156154 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:50.156323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.153887 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:50.156446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.156427 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:50.267076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.267037 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:45:50.267540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.267484 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podStartSLOduration=2.267467647 podStartE2EDuration="2.267467647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:50.064118388 +0000 UTC m=+857.768957728" watchObservedRunningTime="2026-11-03 19:45:50.267467647 +0000 UTC m=+857.972306987" Nov 03 19:45:50.316022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.315989 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:45:50.324971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.322488 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:50.325244 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.325209 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:50.370427 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.370356 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podStartSLOduration=3.370331893 podStartE2EDuration="3.370331893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:50.121605791 +0000 UTC m=+857.826445130" watchObservedRunningTime="2026-11-03 19:45:50.370331893 +0000 UTC m=+858.075171234" Nov 03 19:45:50.648612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.636616 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:45:50.648612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.636747 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:50.648612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.636789 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:50.648612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.638762 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:45:50.648612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.638864 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:50.648612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.638896 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:50.716606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.715680 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:50.848727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.848671 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:50.922787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:50.922683 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.006093 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.006053 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:51.012018 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.011420 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:51.074602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.073974 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:51.074602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.074035 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:51.097004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.096890 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:51.097004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.096965 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:51.135758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.134679 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.135758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.135131 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.135758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.135178 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.135758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.135480 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.316693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.316653 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:51.316976 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.316946 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:51.357095 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.357044 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.404788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.404727 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.639684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.623220 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:51.639684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.623288 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:51.663794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.663754 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:51.686102 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.686063 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:51.746708 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.727644 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.746708 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.728171 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:45:51.871048 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.870994 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.872433 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.871417 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:45:51.987017 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.986840 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:51.988064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:51.987996 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:45:52.019855 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.019823 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:52.020118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.020082 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:52.181823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.115047 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:52.181823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.115108 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:52.181823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.126547 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:52.181823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.126628 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:52.181823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.135669 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:52.397604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.395037 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:52.503616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.438249 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:52.503616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.465979 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:52.503616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.466062 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:52.503616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.499569 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:52.503616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.502814 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:45:52.503616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.502998 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:45:52.503616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.503064 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:45:52.765658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.718284 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:52.765961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.765923 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:52.796481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.794958 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:45:52.908763 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.881433 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:45:52.973517 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:52.973470 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.313817 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.313777 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.322743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.322704 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:53.420606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.321732 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.420606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.374021 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.420606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.374100 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:53.420606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.406628 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:53.461624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.434643 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.461624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.435165 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.461624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.439702 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.461624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.439826 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.461624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.439853 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:53.462544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.462432 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.462544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.462511 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:53.544892 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.544842 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.628797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.628693 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:53.839607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.799028 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.839607 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:53.835626 4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.176s" Nov 03 19:45:53.839607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.835718 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:45:53.841857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.841822 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.842089 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.842062 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:53.881717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.881625 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:53.881959 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.881930 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:53.907792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.905186 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.928382 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:45:53.915930 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:45:53.945799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.941791 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:53.962618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.961723 4314 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-cjcmt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.8:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:45:53.962618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:53.961791 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cjcmt" podUID="4ab98eaf-1317-4f84-aab7-b3a89831ea07" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.8:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:54.006535 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.006498 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:54.006823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.006790 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:54.157033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.156950 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:54.157362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.157330 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:54.167126 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.167100 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:54.170697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.170664 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:54.307739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.305999 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 03 19:45:54.356611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.334425 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:54.356611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.334483 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:54.356611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.343777 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:54.601607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.597716 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:54.601607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.598127 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:45:54.601607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.600751 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:54.602042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.601999 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:54.633270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.633234 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:54.634987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.634945 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:54.635221 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.635195 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:54.635663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.635643 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:54.635821 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.635800 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:54.744481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.744437 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:45:54.878094 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.849712 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:45:54.905620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.905413 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 19:45:54.954975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:54.944657 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:55.040606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.005284 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:55.065607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.040901 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:55.093359 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.093322 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:55.123510 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.123471 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:55.123758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.123156 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:55.123902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.123881 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:55.325607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.315296 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:55.325607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.316720 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:55.325607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.316765 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:55.345715 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.345664 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:55.678661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.666175 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:55.678661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.668169 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:45:55.678661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.668653 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:55.678661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.668694 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:55.678661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.668785 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:55.678661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.669118 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:55.678661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.669473 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:45:55.691863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.690243 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:55.691863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.690291 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:55.890607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.889986 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:55.935201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.931823 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:45:55.935201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.931907 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:55.950607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:55.946842 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:56.008606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.008085 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:56.008606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.008166 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:56.081876 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.074288 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:56.081876 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.074368 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:56.097606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.088006 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:56.097606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.088060 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:56.324610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.313970 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:56.324610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.314033 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:56.363248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.352682 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:56.646669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.628657 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:56.646669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.628728 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:56.669923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.669873 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:56.775606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.670386 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:56.775606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.670948 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:56.775606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.671030 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:56.775606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.767798 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:56.775606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.672513 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:56.775606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.672370 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:56.899761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.898495 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:45:56.950366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:56.950323 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:45:57.074142 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.074092 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:57.074448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.074410 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:57.090066 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.090023 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:57.099837 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.099786 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:57.100517 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.093402 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:57.100735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.100710 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:57.314209 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.314167 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:57.314508 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.314465 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:57.358601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.354723 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.465608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.464795 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:45:57.465608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.464869 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.651456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.631013 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:57.651456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.631074 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:57.651456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.648172 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:57.651456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.648224 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:57.708609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.707680 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:45:57.722601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.710661 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:45:57.722601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.711860 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.767979 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.767933 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.769628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.768237 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:45:57.770937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.768266 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:45:57.770937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.768285 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:45:57.770937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.770867 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.935846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.935307 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:45:57.935846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.935372 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.942053 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.942007 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:57.958244 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:57.958201 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.007164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.007091 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:58.007164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.007150 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:58.083607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.081099 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:58.083607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.081159 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:58.112905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.112868 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:58.113168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.113142 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:58.372606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.368617 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:58.416134 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.414916 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:58.416134 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.414975 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:58.656675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.647218 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:58.656675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.647281 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:58.656675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.649019 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:58.656675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.649083 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:58.739639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.738027 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.772601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.769891 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.773613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.773280 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.809615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.783670 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.809615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.797816 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:45:58.886609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.886174 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:45:58.949733 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.949631 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:58.967618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:58.959159 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:45:59.012641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.012176 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:45:59.012641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.012238 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:45:59.090680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.090541 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.091141 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.090940 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:45:59.091141 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.091095 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:45:59.119062 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.119029 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:45:59.119329 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.119296 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:45:59.337160 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.337120 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:45:59.337511 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.337420 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:45:59.370721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.370675 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:45:59.624660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.624514 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:45:59.624660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.624571 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:45:59.650917 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.650885 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:45:59.651333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.651302 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:45:59.745606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.745237 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.813607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.807380 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.832112 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.819292 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.832112 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.821342 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:45:59.957662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.956687 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:45:59.983607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:45:59.968272 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.005468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.005006 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:46:00.005468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.005073 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:46:00.072323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.071907 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:46:00.072323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.071965 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:46:00.117623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.090315 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:46:00.117623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.090371 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:46:00.350622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.350502 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:46:00.351000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.350870 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:46:00.386170 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.373724 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:00.644075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.643980 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:00.647810 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.647774 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:00.648332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.645344 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:00.648516 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.648495 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:00.764608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.749567 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.812951 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.812906 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.829640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.829497 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:00.831540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.829903 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:00.831540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.829922 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:00.957773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.957661 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:00.980605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:00.971059 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:01.007398 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.006988 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:46:01.007398 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.007047 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:46:01.081661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.070909 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:46:01.081661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.070971 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:46:01.090028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.089913 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:46:01.090028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.089969 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:46:01.346074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.345363 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:46:01.346074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.345442 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:46:01.352101 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.351091 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 19:46:01.425975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.425919 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:01.428274 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.428221 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.723671 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.719487 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:01.732887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.732846 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:01.839610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.817762 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.963615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.818806 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.964111 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.839952 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.967603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.840004 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.967603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.839917 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:01.967603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:01.967416 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:46:02.007618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.004393 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:02.174164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.168651 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:46:02.174164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.168717 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:46:02.398451 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.398353 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:46:02.399275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.399235 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:46:02.399522 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.399506 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:46:02.399668 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.399651 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:46:02.433775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.433721 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.828525 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.828474 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:02.828816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.828787 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:02.924728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.924671 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.950605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.948698 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:02.966730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.966680 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:02.967478 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.967443 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:02.971602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:02.967721 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:03.018607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.017655 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.018607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.017732 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:03.043609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.033275 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:03.101701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.100988 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.101701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.101047 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:03.139608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.122708 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.139608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.122763 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:46:03.147269 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.147231 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:03.203624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.203565 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.203903 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.203859 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:46:03.359666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.357004 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.359666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.357091 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:46:03.359666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.357487 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.359666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.357529 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:46:03.359666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.357643 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.359666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.357686 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:46:03.459844 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.446998 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:03.527053 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.527013 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:46:03.527331 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.527304 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:46:03.557144 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.557108 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:46:03.557415 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.557391 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:46:03.558322 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.555724 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:03.660695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.655686 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:46:03.664405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.662131 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:46:03.730079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.702258 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.730079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.707061 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:03.730079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.707116 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:03.756636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.751799 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:03.927768 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.927660 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:03.996937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.969790 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:03.996937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.970450 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:03.996937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:03.969736 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:04.041606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.023678 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:04.066638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.050055 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:46:04.066638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.050119 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:46:04.066638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.050460 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:04.158365 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.158309 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:04.183544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.183451 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:46:04.196483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.188853 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:46:04.236992 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.201611 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:46:04.236992 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.201694 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:46:04.388620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.323661 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:46:04.388620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.323720 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:46:04.388620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.341978 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:04.453429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.453307 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:04.647775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.647728 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:04.648055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.648013 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:04.655609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.648764 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:04.655863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.655826 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:04.819505 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.819473 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:46:04.854602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.853862 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.014701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.941689 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:46:05.014701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.974664 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.014701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:04.996741 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:05.014701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.006524 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:46:05.014701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.006575 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:46:05.039679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.028462 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.071755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.071638 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.139630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.132448 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:46:05.139630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.132512 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:46:05.139630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.132630 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:46:05.139630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.132659 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:46:05.166672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.166620 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:05.303192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.303137 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.332834 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.332738 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:46:05.333103 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.333071 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:46:05.481607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.456858 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.632314 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.632231 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:05.657727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.657683 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:05.659769 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.659740 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:05.659949 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.659925 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:05.818615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.798977 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.952803 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.952702 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:05.992942 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:05.992899 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:06.002816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.002777 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:06.018647 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.011031 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 19:46:06.055484 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.055444 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 19:46:06.061819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.033660 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:06.090940 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.071379 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 19:46:06.091157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.091125 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 19:46:06.116614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.108400 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 19:46:06.117045 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.117008 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 19:46:06.117497 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.117461 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:06.238788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.238693 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:06.372034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.371996 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:46:06.380533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.380494 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:46:06.524996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.523966 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:06.622165 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.622130 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:06.622478 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.622429 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:06.635114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.635085 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:06.635310 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.635285 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:06.827958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.826746 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:06.827958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:06.827287 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:46:07.067605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.056306 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:07.067605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.057936 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:07.067605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.059285 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:46:07.107593 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.103106 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:07.131346 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.130683 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:07.247743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.247689 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:07.551603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.547690 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:07.621605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.620894 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:46:07.644264 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.632486 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:46:07.644264 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.632534 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:46:07.644611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.644570 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:46:07.644735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.644715 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:46:07.706602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.704413 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 19:46:07.820655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.818452 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:46:07.852492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.849371 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:46:07.852492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.850264 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:07.852492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.851509 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:07.895602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.893888 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:46:07.904317 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:07.904279 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:46:08.019615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.002650 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:46:08.019615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.014836 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:46:08.101795 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.101621 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:08.112866 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.105615 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:08.112866 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.105667 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:08.150765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.150101 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:46:08.187720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.180253 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 19:46:08.187720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.180816 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:08.311189 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.309531 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:08.563928 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.562500 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:08.723500 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.723465 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:46:08.852742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.735920 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:46:08.852742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.771262 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:46:08.852742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.843274 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:08.852742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.845927 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:46:08.924177 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:08.924134 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:09.127653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.125198 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:09.166654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.163775 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:09.257644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.254917 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:09.288544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.288512 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:46:09.291937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.291882 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:46:09.317128 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.316916 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:09.505262 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.502352 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:46:09.543923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.542724 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:46:09.599076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.599048 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:46:09.626801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:09.625943 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:46:17.653744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.653696 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv"] Nov 03 19:46:17.678014 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.677799 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv"] Nov 03 19:46:17.678014 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.678011 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704135 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704339 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704198 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704339 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704229 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704339 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704274 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbdd\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-kube-api-access-qnbdd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704505 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704393 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704505 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704433 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704505 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704483 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704517 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.704679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.704546 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ea0ebd-b1b0-4075-863b-0794864e0c24-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.806577 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806543 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.806577 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806593 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ea0ebd-b1b0-4075-863b-0794864e0c24-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.806827 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806769 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.806827 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806802 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.806933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806829 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.806933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806870 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbdd\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-kube-api-access-qnbdd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807041 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806962 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807041 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.806972 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807041 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.807023 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.807070 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807252 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.807222 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.807324 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ea0ebd-b1b0-4075-863b-0794864e0c24-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807514 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.807482 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.807680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.807658 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.809622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.809602 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.809971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.809949 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.815014 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.814988 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbdd\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-kube-api-access-qnbdd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:17.815119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:17.815103 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:18.015671 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.015638 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:18.182644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.178204 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k"] Nov 03 19:46:18.194621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.193510 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k"] Nov 03 19:46:18.194621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.193670 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.215975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.215948 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216093 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.215992 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216093 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.216051 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4vp\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-kube-api-access-nm4vp\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.216091 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ed5324-5bf5-4535-9fe6-042276f0e767-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216255 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.216237 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216352 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.216338 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.216394 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216442 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.216424 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.216471 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.216454 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.317676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.317603 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.317830 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.317679 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.317830 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.317719 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.317830 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.317736 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.317830 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.317763 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.317843 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.317861 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.318057 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4vp\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-kube-api-access-nm4vp\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.318121 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ed5324-5bf5-4535-9fe6-042276f0e767-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.318130 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.318175 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318296 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.318200 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.318794 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ed5324-5bf5-4535-9fe6-042276f0e767-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.318869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.318841 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.319852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.319822 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.320127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.320104 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.329685 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.329658 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.330306 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.330273 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4vp\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-kube-api-access-nm4vp\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.352864 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.352842 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv"] Nov 03 19:46:18.353699 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:46:18.353675 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ea0ebd_b1b0_4075_863b_0794864e0c24.slice/crio-bb5ee4e3c87f3dba43942eb69d5cca966c647bd23108c1fc322379c21f71b9dd WatchSource:0}: Error finding container bb5ee4e3c87f3dba43942eb69d5cca966c647bd23108c1fc322379c21f71b9dd: Status 404 returned error can't find the container with id bb5ee4e3c87f3dba43942eb69d5cca966c647bd23108c1fc322379c21f71b9dd Nov 03 19:46:18.356348 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.356307 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:18.356423 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.356382 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:18.356423 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.356411 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:18.518155 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.518112 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:18.824058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.823646 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k"] Nov 03 19:46:18.824980 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:46:18.824953 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ed5324_5bf5_4535_9fe6_042276f0e767.slice/crio-b5c73ce328dad9670cd9abb956f9777543585eea968bd74c59d764cec78dc22a WatchSource:0}: Error finding container b5c73ce328dad9670cd9abb956f9777543585eea968bd74c59d764cec78dc22a: Status 404 returned error can't find the container with id b5c73ce328dad9670cd9abb956f9777543585eea968bd74c59d764cec78dc22a Nov 03 19:46:18.829014 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.828977 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:18.829122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.829047 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:18.829122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:18.829083 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:19.152734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.152620 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" event={"ID":"c4ed5324-5bf5-4535-9fe6-042276f0e767","Type":"ContainerStarted","Data":"0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f"} Nov 03 19:46:19.152892 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.152751 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" event={"ID":"c4ed5324-5bf5-4535-9fe6-042276f0e767","Type":"ContainerStarted","Data":"b5c73ce328dad9670cd9abb956f9777543585eea968bd74c59d764cec78dc22a"} Nov 03 19:46:19.154735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.154703 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" event={"ID":"98ea0ebd-b1b0-4075-863b-0794864e0c24","Type":"ContainerStarted","Data":"ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2"} Nov 03 19:46:19.154833 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.154758 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" event={"ID":"98ea0ebd-b1b0-4075-863b-0794864e0c24","Type":"ContainerStarted","Data":"bb5ee4e3c87f3dba43942eb69d5cca966c647bd23108c1fc322379c21f71b9dd"} Nov 03 19:46:19.187652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.185072 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podStartSLOduration=1.185051573 podStartE2EDuration="1.185051573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.176936466 +0000 UTC m=+886.881775807" watchObservedRunningTime="2026-11-03 19:46:19.185051573 +0000 UTC m=+886.889890915" Nov 03 19:46:19.211873 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.211810 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podStartSLOduration=2.21179058 podStartE2EDuration="2.21179058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.2100679 +0000 UTC m=+886.914907252" watchObservedRunningTime="2026-11-03 19:46:19.21179058 +0000 UTC m=+886.916629921" Nov 03 19:46:19.519334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.519297 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:19.520738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.520711 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:19.520869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:19.520767 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:20.017701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:20.017669 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:20.018254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:20.018224 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:20.018401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:20.018382 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:20.523057 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:20.523013 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:20.523290 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:20.523077 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:21.020607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:21.019006 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:21.020607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:21.019081 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:21.523613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:21.523398 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:21.523613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:21.523457 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:22.017071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:22.017034 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:22.017368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:22.017327 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:22.041983 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:46:22.041952 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:46:22.521118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:22.521064 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:22.521405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:22.521361 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:23.016351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:23.016315 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:23.016656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:23.016625 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:23.737604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:23.734782 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:23.737604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:23.734858 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:24.121497 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.119028 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:24.121497 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.119100 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:24.464684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.444501 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:24.522721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.522241 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:24.522721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.522304 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:24.741714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.737647 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:24.892415 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.891016 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:24.894360 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.892694 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 19:46:24.894360 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.892769 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:24.895662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.894602 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:24.946706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:24.946658 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:25.018659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.018245 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:25.018659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.018325 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:25.035833 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.035788 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:25.323293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.321175 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 19:46:25.385286 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.384705 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:25.447055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.447011 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:25.447821 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.447798 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:46:25.542696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.542661 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:25.571243 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.571188 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:25.801607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.801002 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:25.895608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.836040 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:46:25.945663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.922661 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:25.945663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.923335 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:25.945663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.923988 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:25.945663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.924707 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:46:25.945663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.924964 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:46:25.945663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:25.925040 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:46:26.099608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.041196 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:26.099608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.041544 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:26.099608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.050897 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:26.128606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.128167 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:26.184363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.129060 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:46:26.184363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.129205 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:46:26.420604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.386891 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:26.520615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.463560 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:46:26.729878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.729776 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:26.739613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.730885 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:26.838609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.809721 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" podUID="0067d07b-adb9-4976-a4fb-6bdcec9091cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:26.840202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.840158 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:26.872819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.872774 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:26.937657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.924719 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:26.972608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.937754 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:27.051641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:26.937818 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:27.280606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.275828 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:27.280606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.276957 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:27.292607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.287655 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:27.292607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.287712 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:27.478666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.455676 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:27.587607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.563720 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:27.587607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.563801 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:27.918607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.857512 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:27.918607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.862236 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Nov 03 19:46:27.918607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.874094 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:46:27.955469 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:27.948251 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.003343 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.003422 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.048992 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.050011 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.050057 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.055150 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.055205 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:28.174665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.055280 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:28.303818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.303770 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:28.352165 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.303721 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:28.458788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.458681 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:28.522237 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.522200 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:28.522508 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.522474 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:28.904609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.860429 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:28.904971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.879697 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:46:28.957720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:28.957672 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:29.032430 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.032395 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:29.032742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.032709 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:29.060609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.056680 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.060609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.057057 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:29.130611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.129672 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.364605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.363733 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.364605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.364164 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.476810 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.475721 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.521984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.521943 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:29.522254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.522223 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:29.917607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.913795 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.938606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.936334 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:29.967607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:29.966762 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:30.019038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.018997 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 19:46:30.019343 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.019308 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 19:46:30.124607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.122046 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:30.124607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.122390 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:30.409602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.382754 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:30.409602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.383799 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.499610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.487564 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.519128 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.519057 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:30.519433 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.519380 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:30.551721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.551684 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:30.552019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.551978 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:30.941114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.941066 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:31.004607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.982959 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:31.004607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:30.985422 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:31.085025 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.084990 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:46:31.149616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.145708 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:31.154612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.150263 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:31.418648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.418597 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:31.419176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.419149 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 19:46:31.441030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.440990 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 03 19:46:31.489761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.489708 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:31.521456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.521421 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:46:31.524985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.524933 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 19:46:31.525206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.525184 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 19:46:31.633199 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.633165 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:46:31.634925 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.634901 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:46:31.640032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.640002 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:46:31.640637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.640613 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:46:31.643150 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.643130 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:46:31.643287 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.643277 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:46:31.753045 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.753004 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:31.754052 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.754032 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:31.758905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:31.758886 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:46:32.664992 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:32.664966 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:46:32.773162 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:32.773132 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:46:33.187784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:33.187696 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:33.188907 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:33.188866 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:33.191612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:33.191575 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 19:46:38.147024 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.146989 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7105-841a33"] Nov 03 19:46:38.153177 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.153150 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:38.157090 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.157056 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7105-841a33"] Nov 03 19:46:38.157436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.157370 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bntlr\"" Nov 03 19:46:38.217966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.217921 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbs5\" (UniqueName: \"kubernetes.io/projected/42a95c86-ee93-4224-ad49-4212bbff5c69-kube-api-access-ddbs5\") pod \"test-curl-7105-841a33\" (UID: \"42a95c86-ee93-4224-ad49-4212bbff5c69\") " pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:38.319856 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.319826 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbs5\" (UniqueName: \"kubernetes.io/projected/42a95c86-ee93-4224-ad49-4212bbff5c69-kube-api-access-ddbs5\") pod \"test-curl-7105-841a33\" (UID: \"42a95c86-ee93-4224-ad49-4212bbff5c69\") " pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:38.327704 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.327674 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbs5\" (UniqueName: \"kubernetes.io/projected/42a95c86-ee93-4224-ad49-4212bbff5c69-kube-api-access-ddbs5\") pod \"test-curl-7105-841a33\" (UID: \"42a95c86-ee93-4224-ad49-4212bbff5c69\") " pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:38.481500 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.481410 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:38.768761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.768726 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7105-841a33"] Nov 03 19:46:38.768761 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:46:38.768748 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42a95c86_ee93_4224_ad49_4212bbff5c69.slice/crio-d67f05ae6de3e2d13cb820ca19329c4ba7ffdf75c19d5c972723415617fc7cdb WatchSource:0}: Error finding container d67f05ae6de3e2d13cb820ca19329c4ba7ffdf75c19d5c972723415617fc7cdb: Status 404 returned error can't find the container with id d67f05ae6de3e2d13cb820ca19329c4ba7ffdf75c19d5c972723415617fc7cdb Nov 03 19:46:38.770845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:38.770827 4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:46:39.572990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:39.572951 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7105-841a33" event={"ID":"42a95c86-ee93-4224-ad49-4212bbff5c69","Type":"ContainerStarted","Data":"d67f05ae6de3e2d13cb820ca19329c4ba7ffdf75c19d5c972723415617fc7cdb"} Nov 03 19:46:40.584476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:40.584426 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7105-841a33" event={"ID":"42a95c86-ee93-4224-ad49-4212bbff5c69","Type":"ContainerStarted","Data":"a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db"} Nov 03 19:46:40.600198 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:40.600131 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7105-841a33" podStartSLOduration=1.68113934 podStartE2EDuration="2.600111958s" totalImagesPullingTime="918.972618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:40.598565969 +0000 UTC m=+908.303405310" watchObservedRunningTime="2026-11-03 19:46:40.600111958 +0000 UTC m=+908.304951299" Nov 03 19:46:41.319758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.319721 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7105-841a33"] Nov 03 19:46:41.325019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.324991 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7105-841a33"] Nov 03 19:46:41.461605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.457906 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7105-72a74b"] Nov 03 19:46:41.461605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.459125 4314 cpu_manager.go:414] "containerMap: removing container" podUID="42a95c86-ee93-4224-ad49-4212bbff5c69" containerName="test-curl-7105-841a33" Nov 03 19:46:41.461605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.459149 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a95c86-ee93-4224-ad49-4212bbff5c69" containerName="test-curl-7105-841a33" Nov 03 19:46:41.461605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.459308 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a95c86-ee93-4224-ad49-4212bbff5c69" containerName="test-curl-7105-841a33" Nov 03 19:46:41.466629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.465167 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:41.472101 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.471450 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7105-72a74b"] Nov 03 19:46:41.573667 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.572894 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrnb\" (UniqueName: \"kubernetes.io/projected/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71-kube-api-access-tzrnb\") pod \"test-curl-7105-72a74b\" (UID: \"28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71\") " pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:41.675396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.675361 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrnb\" (UniqueName: \"kubernetes.io/projected/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71-kube-api-access-tzrnb\") pod \"test-curl-7105-72a74b\" (UID: \"28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71\") " pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:41.683702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.683671 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrnb\" (UniqueName: \"kubernetes.io/projected/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71-kube-api-access-tzrnb\") pod \"test-curl-7105-72a74b\" (UID: \"28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71\") " pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:41.788136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:41.788100 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:42.080639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:42.079916 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7105-72a74b"] Nov 03 19:46:42.083992 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:46:42.083958 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28f28c46_b6ce_4a3d_b0e5_7ac0927f2a71.slice/crio-332f4d3f25710d0669f39a1c8b7828e7587597d91712ce3b46af934905a1e0b2 WatchSource:0}: Error finding container 332f4d3f25710d0669f39a1c8b7828e7587597d91712ce3b46af934905a1e0b2: Status 404 returned error can't find the container with id 332f4d3f25710d0669f39a1c8b7828e7587597d91712ce3b46af934905a1e0b2 Nov 03 19:46:42.602419 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:42.602381 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7105-72a74b" event={"ID":"28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71","Type":"ContainerStarted","Data":"3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e"} Nov 03 19:46:42.602659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:42.602432 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7105-72a74b" event={"ID":"28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71","Type":"ContainerStarted","Data":"332f4d3f25710d0669f39a1c8b7828e7587597d91712ce3b46af934905a1e0b2"} Nov 03 19:46:42.602760 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:42.602725 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7105-841a33" podUID="42a95c86-ee93-4224-ad49-4212bbff5c69" containerName="test-curl-7105-841a33" containerID="cri-o://a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db" gracePeriod=2 Nov 03 19:46:42.622328 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:42.622284 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7105-72a74b" podStartSLOduration=1.6222706009999999 podStartE2EDuration="1.622270601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:42.618536497 +0000 UTC m=+910.323375838" watchObservedRunningTime="2026-11-03 19:46:42.622270601 +0000 UTC m=+910.327109941" Nov 03 19:46:43.325364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:43.325328 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7105-72a74b"] Nov 03 19:46:43.333603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:43.332323 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7105-72a74b"] Nov 03 19:46:44.620563 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:44.620505 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7105-72a74b" podUID="28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" containerName="test-curl-7105-72a74b" containerID="cri-o://3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e" gracePeriod=2 Nov 03 19:46:44.849257 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:44.849225 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:44.928298 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:44.928217 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a95c86-ee93-4224-ad49-4212bbff5c69-kube-api-access-ddbs5\" (UniqueName: \"kubernetes.io/projected/42a95c86-ee93-4224-ad49-4212bbff5c69-kube-api-access-ddbs5\") pod \"42a95c86-ee93-4224-ad49-4212bbff5c69\" (UID: \"42a95c86-ee93-4224-ad49-4212bbff5c69\") " Nov 03 19:46:44.930301 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:44.930272 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a95c86-ee93-4224-ad49-4212bbff5c69-kube-api-access-ddbs5" pod "42a95c86-ee93-4224-ad49-4212bbff5c69" (UID: "42a95c86-ee93-4224-ad49-4212bbff5c69"). InnerVolumeSpecName "kube-api-access-ddbs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:46:45.030115 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:45.030082 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddbs5\" (UniqueName: \"kubernetes.io/projected/42a95c86-ee93-4224-ad49-4212bbff5c69-kube-api-access-ddbs5\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:45.628193 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:45.628152 4314 generic.go:356] "Generic (PLEG): container finished" podID="42a95c86-ee93-4224-ad49-4212bbff5c69" containerID="a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db" exitCode=137 Nov 03 19:46:45.628664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:45.628192 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:45.628664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:45.628243 4314 scope.go:122] "RemoveContainer" containerID="a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db" Nov 03 19:46:45.639043 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:45.639027 4314 scope.go:122] "RemoveContainer" containerID="a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db" Nov 03 19:46:45.639408 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:46:45.639386 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db\": container with ID starting with a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db not found: ID does not exist" containerID="a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db" Nov 03 19:46:45.639486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:45.639417 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db"} err="failed to get container status \"a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db\": rpc error: code = NotFound desc = could not find container \"a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db\": container with ID starting with a8940d02d940a1e053312bbd3e94a5616628f266aabf75064fbef385e7db73db not found: ID does not exist" Nov 03 19:46:45.641049 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:46:45.641021 4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7105-841a33\" is forbidden: User \"system:node:ip-10-0-133-204.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-204.us-east-2.compute.internal' and this object" podUID="42a95c86-ee93-4224-ad49-4212bbff5c69" pod="opendatahub/test-curl-7105-841a33" Nov 03 19:46:46.429471 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:46.429438 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a95c86-ee93-4224-ad49-4212bbff5c69" path="/var/lib/kubelet/pods/42a95c86-ee93-4224-ad49-4212bbff5c69/volumes" Nov 03 19:46:46.898372 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:46.898347 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:46.971388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:46.971359 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71-kube-api-access-tzrnb\" (UniqueName: \"kubernetes.io/projected/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71-kube-api-access-tzrnb\") pod \"28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71\" (UID: \"28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71\") " Nov 03 19:46:46.973408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:46.973375 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71-kube-api-access-tzrnb" pod "28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" (UID: "28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71"). InnerVolumeSpecName "kube-api-access-tzrnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:46:47.074226 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:47.074193 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrnb\" (UniqueName: \"kubernetes.io/projected/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71-kube-api-access-tzrnb\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:46:47.649200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:47.649154 4314 generic.go:356] "Generic (PLEG): container finished" podID="28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" containerID="3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e" exitCode=137 Nov 03 19:46:47.649408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:47.649211 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:47.649408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:47.649246 4314 scope.go:122] "RemoveContainer" containerID="3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e" Nov 03 19:46:47.660234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:47.660062 4314 scope.go:122] "RemoveContainer" containerID="3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e" Nov 03 19:46:47.660413 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:46:47.660393 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e\": container with ID starting with 3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e not found: ID does not exist" containerID="3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e" Nov 03 19:46:47.660478 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:47.660425 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e"} err="failed to get container status \"3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e\": rpc error: code = NotFound desc = could not find container \"3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e\": container with ID starting with 3521b5dabf407532f83dee22de575643f24aab97eed133451b8a0a2f0ccff01e not found: ID does not exist" Nov 03 19:46:47.662064 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:46:47.662023 4314 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7105-72a74b\" is forbidden: User \"system:node:ip-10-0-133-204.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-204.us-east-2.compute.internal' and this object" podUID="28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" pod="opendatahub/test-curl-7105-72a74b" Nov 03 19:46:48.430796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:48.430758 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" path="/var/lib/kubelet/pods/28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71/volumes" Nov 03 19:46:51.998091 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:46:51.998050 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:46:52.289575 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.289494 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:46:52.954065 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.954027 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn"] Nov 03 19:46:52.955154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.955126 4314 cpu_manager.go:414] "containerMap: removing container" podUID="28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" containerName="test-curl-7105-72a74b" Nov 03 19:46:52.955154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.955153 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" containerName="test-curl-7105-72a74b" Nov 03 19:46:52.955382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.955365 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f28c46-b6ce-4a3d-b0e5-7ac0927f2a71" containerName="test-curl-7105-72a74b" Nov 03 19:46:52.959860 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.959837 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:52.969901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.969878 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cbfcf47a-openshift-default-dockercfg-xgjf5\"" Nov 03 19:46:52.979791 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:52.979766 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn"] Nov 03 19:46:53.082950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.082913 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a093b1d-8e0e-4703-8aac-149aae6db744-istiod-ca-cert\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.082950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.082949 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8g4s\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-kube-api-access-s8g4s\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.083366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.083073 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-data\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.083366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.083150 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-certs\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.083366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.083182 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-socket\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.083366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.083272 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-envoy\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.083366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.083341 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-podinfo\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.083366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.083357 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-token\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.083542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.083402 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-credential-socket\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.184702 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.184662 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-podinfo\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.184921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.184712 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-token\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.184921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.184763 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-credential-socket\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.184921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.184835 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a093b1d-8e0e-4703-8aac-149aae6db744-istiod-ca-cert\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.184921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.184866 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8g4s\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-kube-api-access-s8g4s\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.185143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.184951 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-data\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.185143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.185084 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-certs\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.185143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.185133 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-socket\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.185625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.185561 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-data\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.185824 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.185800 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-credential-socket\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.185967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.185943 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-envoy\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.186346 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.186288 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-socket\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.186673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.186653 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-certs\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.187144 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.187120 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a093b1d-8e0e-4703-8aac-149aae6db744-istiod-ca-cert\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.188750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.188719 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-podinfo\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.188934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.188914 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-envoy\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.195231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.195183 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-token\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.202603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.195506 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8g4s\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-kube-api-access-s8g4s\") pod \"e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.296773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.296741 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:53.603444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.603359 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn"] Nov 03 19:46:53.613058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.612622 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:53.613058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.612711 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:53.613058 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.612746 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:46:53.745378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:53.745338 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" event={"ID":"6a093b1d-8e0e-4703-8aac-149aae6db744","Type":"ContainerStarted","Data":"5708b91c77491a14234f133c4d9254e7d132883cec0460e0cafdbffe3c96d7b9"} Nov 03 19:46:54.827613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:54.816617 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" event={"ID":"6a093b1d-8e0e-4703-8aac-149aae6db744","Type":"ContainerStarted","Data":"dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d"} Nov 03 19:46:54.851607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:54.847823 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:54.851607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:54.848423 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:54.851607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:54.849952 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" podStartSLOduration=2.849935535 podStartE2EDuration="2.849935535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:54.848365382 +0000 UTC m=+922.553204723" watchObservedRunningTime="2026-11-03 19:46:54.849935535 +0000 UTC m=+922.554774876" Nov 03 19:46:54.881608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:54.880153 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:54.955622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:54.952667 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:55.087665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.069108 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:55.185610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.159796 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:55.292608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.279923 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:55.292608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.280349 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:55.308013 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.307975 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:55.311720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.311676 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:55.319617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.319568 4314 patch_prober.go:29] interesting pod/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 19:46:55.319893 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.319863 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 19:46:55.476966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.475923 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:55.867639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.849323 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:55.867639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.849914 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:46:55.914618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.889755 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:55.914618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.890397 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:46:56.010610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:55.971314 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:56.010610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.005873 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:46:56.078990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.078944 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:56.110607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.083172 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:46:56.238327 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.238227 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:56.324596 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.323941 4314 patch_prober.go:29] interesting pod/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 19:46:56.324596 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.323989 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 19:46:56.324596 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.291048 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:56.324940 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.289839 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:56.331651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.330069 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:46:56.339748 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.339724 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:46:56.339994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.339963 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:46:56.483670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.483622 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:56.716491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.605569 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:46:56.870609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.855865 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:56.909293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.908964 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:56.996731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:56.996313 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:57.105857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.087715 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:57.357665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.346701 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:57.357665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.347078 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:57.357665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.348382 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:57.447924 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.446935 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:57.484409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.484349 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:57.681814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.681705 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:57.682073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.682032 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:57.827607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.801707 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:57.827607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.801785 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:57.912926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.912872 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:57.919819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.919768 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:46:57.943903 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:57.925282 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Nov 03 19:46:58.053626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.045733 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:58.053626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.045808 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:58.105643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.098676 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:58.109613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.106908 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:58.194037 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.193915 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:46:58.194316 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.194265 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:58.377617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.361694 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:58.445684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.378632 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:58.445684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.379024 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:46:58.622853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.609786 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:46:58.955731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.931800 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:58.955731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:58.932164 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:46:59.013610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.006296 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:59.038768 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.038728 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:46:59.165610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.161729 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:46:59.256674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.252983 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 19:46:59.256674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.253304 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:46:59.428476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.428254 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:59.435054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.434995 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:46:59.435671 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.435643 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:46:59.673675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.658810 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:59.932816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.932698 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:46:59.934547 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:46:59.933058 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:47:00.168607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.167731 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:47:00.260765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.260711 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:47:00.444691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.444646 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:47:00.445309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.445275 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:47:00.445997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.445294 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:47:00.581827 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.581710 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:00.582120 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.582068 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:00.638167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.638039 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:00.638567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.638443 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:00.665608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.663696 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:00.960664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.935685 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:00.960664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:00.936087 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.187757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.187700 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.267611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.266391 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.356750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.356695 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 19:47:01.395722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.395668 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.453729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.453677 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.454381 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.454340 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:47:01.455849 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.455822 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.665770 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.664691 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.669719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.669684 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:01.671099 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.671069 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.936850 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.936741 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:01.937695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:01.937116 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:47:02.002738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.002699 4314 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:02.003012 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.002980 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.054030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.053676 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:02.054030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.053742 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.402635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.318745 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.402635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.349874 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:47:02.540303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.480835 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.540303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.481201 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:47:02.540303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.481485 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:47:02.680930 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.680826 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:47:02.799773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.799729 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:02.800053 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.800020 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.939799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.939692 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:02.940791 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:02.940755 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.349819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.349336 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.353180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.352067 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.405278 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.405033 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.535521 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.519915 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.537754 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.535872 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:47:03.537754 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.537057 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:47:03.685681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.685150 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.967686 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.965777 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:03.967686 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:03.966152 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:04.302100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.301796 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 03 19:47:04.350760 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.349762 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:04.357132 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.354487 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:04.536742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.536696 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:04.545666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.545055 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:04.583191 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.583112 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:47:04.591764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.591729 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:47:04.678019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.677988 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:47:04.687371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.687345 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:47:04.722630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.710366 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:04.823228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.823201 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:47:04.823491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.823473 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:47:04.826453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.826436 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:47:04.839147 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.839094 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:47:04.862283 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.862256 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp"] Nov 03 19:47:04.871528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.871507 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:04.892667 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.892638 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cbfcf47a\"" Nov 03 19:47:04.892859 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.892834 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp"] Nov 03 19:47:04.962330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.962292 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbksm\" (UniqueName: \"kubernetes.io/projected/7929582c-f39d-4d78-b8f8-2342bbefeb29-kube-api-access-rbksm\") pod \"payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:04.962546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:04.962521 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7929582c-f39d-4d78-b8f8-2342bbefeb29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:05.064953 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:05.064916 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbksm\" (UniqueName: \"kubernetes.io/projected/7929582c-f39d-4d78-b8f8-2342bbefeb29-kube-api-access-rbksm\") pod \"payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:05.065144 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:05.065031 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7929582c-f39d-4d78-b8f8-2342bbefeb29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:05.065946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:05.065918 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7929582c-f39d-4d78-b8f8-2342bbefeb29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:05.076312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:05.076276 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbksm\" (UniqueName: \"kubernetes.io/projected/7929582c-f39d-4d78-b8f8-2342bbefeb29-kube-api-access-rbksm\") pod \"payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:05.191111 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:05.189864 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:05.746627 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:47:05.746591 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7929582c_f39d_4d78_b8f8_2342bbefeb29.slice/crio-33fc1b19da4d0acfd2a10fffa46ef3007305eb18c4437839bfe1eedccf0303c5 WatchSource:0}: Error finding container 33fc1b19da4d0acfd2a10fffa46ef3007305eb18c4437839bfe1eedccf0303c5: Status 404 returned error can't find the container with id 33fc1b19da4d0acfd2a10fffa46ef3007305eb18c4437839bfe1eedccf0303c5 Nov 03 19:47:05.747870 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:05.747845 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp"] Nov 03 19:47:06.261007 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:06.260972 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" event={"ID":"7929582c-f39d-4d78-b8f8-2342bbefeb29","Type":"ContainerStarted","Data":"b3155e546d7dd06095f9f7bd8d92d75b6892cbfe9505a39d98f9dbd988fa4c14"} Nov 03 19:47:06.261216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:06.261032 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" event={"ID":"7929582c-f39d-4d78-b8f8-2342bbefeb29","Type":"ContainerStarted","Data":"33fc1b19da4d0acfd2a10fffa46ef3007305eb18c4437839bfe1eedccf0303c5"} Nov 03 19:47:06.261278 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:06.261244 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:06.279081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:06.279030 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" podStartSLOduration=2.279016596 podStartE2EDuration="2.279016596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:06.278217889 +0000 UTC m=+933.983057228" watchObservedRunningTime="2026-11-03 19:47:06.279016596 +0000 UTC m=+933.983855935" Nov 03 19:47:07.579986 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.578834 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk"] Nov 03 19:47:07.589118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.587940 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.596052 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.596021 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk"] Nov 03 19:47:07.604037 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.604012 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-26c00c22-openshift-default-dockercfg-svc7d\"" Nov 03 19:47:07.758842 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.758809 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-podinfo\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.758988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.758861 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-socket\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.758988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.758965 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-data\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.759107 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.758991 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fmtx\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-kube-api-access-5fmtx\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.759107 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.759008 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-token\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.759224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.759194 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-envoy\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.759311 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.759295 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-certs\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.759370 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.759345 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istiod-ca-cert\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.759424 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.759375 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-credential-socket\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861309 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-podinfo\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861378 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-socket\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861456 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-data\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861495 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fmtx\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-kube-api-access-5fmtx\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861523 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-token\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861650 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-envoy\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861907 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861738 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-certs\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861907 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861792 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istiod-ca-cert\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.861907 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.861818 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-credential-socket\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.862474 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.862443 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-credential-socket\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.862474 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.862462 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-socket\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.862747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.862722 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-data\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.863297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.863269 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-certs\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.864410 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.864035 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istiod-ca-cert\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.866175 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.866150 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-envoy\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.868090 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.868063 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-podinfo\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.873415 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.873234 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-token\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.884466 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.884337 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fmtx\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-kube-api-access-5fmtx\") pod \"e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:07.925268 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:07.925235 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:08.522217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:08.522176 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk"] Nov 03 19:47:08.540634 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:47:08.540554 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bba2368_e9a4_4ea6_8787_3e3711f5da2f.slice/crio-f2a95608f9bfbe0e2b06da110d2022a985507fc94bf38f77607aa9629af9211f WatchSource:0}: Error finding container f2a95608f9bfbe0e2b06da110d2022a985507fc94bf38f77607aa9629af9211f: Status 404 returned error can't find the container with id f2a95608f9bfbe0e2b06da110d2022a985507fc94bf38f77607aa9629af9211f Nov 03 19:47:08.543323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:08.543278 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:08.543436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:08.543373 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:08.543436 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:08.543417 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:09.348121 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:09.347985 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" event={"ID":"9bba2368-e9a4-4ea6-8787-3e3711f5da2f","Type":"ContainerStarted","Data":"9d9932792c954ce2e4cb7522336fbe9c3d778f34aaacccd34cf28b11710e7dea"} Nov 03 19:47:09.348121 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:09.348058 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" event={"ID":"9bba2368-e9a4-4ea6-8787-3e3711f5da2f","Type":"ContainerStarted","Data":"f2a95608f9bfbe0e2b06da110d2022a985507fc94bf38f77607aa9629af9211f"} Nov 03 19:47:09.372193 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:09.372137 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" podStartSLOduration=2.37212338 podStartE2EDuration="2.37212338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:09.368834368 +0000 UTC m=+937.073673710" watchObservedRunningTime="2026-11-03 19:47:09.37212338 +0000 UTC m=+937.076962719" Nov 03 19:47:09.925469 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:09.925437 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:09.931742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:09.931718 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:10.359599 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:10.359488 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:10.361682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:10.361655 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:47:12.125967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.125928 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4"] Nov 03 19:47:12.130935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.130913 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.138082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.138057 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-26c00c22\"" Nov 03 19:47:12.138200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.138165 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-26c00c22-dockercfg-4nmkx\"" Nov 03 19:47:12.140716 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.140688 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4"] Nov 03 19:47:12.184894 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.184865 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6"] Nov 03 19:47:12.189318 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.189293 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:12.198511 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.198482 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx82k\" (UniqueName: \"kubernetes.io/projected/85dcd066-f5e5-42ef-afa2-b109b5f183dc-kube-api-access-cx82k\") pod \"payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.202275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.202246 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dcd066-f5e5-42ef-afa2-b109b5f183dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.208315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.208294 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6"] Nov 03 19:47:12.449672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.442754 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dcd066-f5e5-42ef-afa2-b109b5f183dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.449672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.442910 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4d93e2-3440-4e06-ac18-90abf97564c2-plugins-config-volume\") pod \"payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:12.449672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.443059 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx82k\" (UniqueName: \"kubernetes.io/projected/85dcd066-f5e5-42ef-afa2-b109b5f183dc-kube-api-access-cx82k\") pod \"payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.449672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.443181 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bc4\" (UniqueName: \"kubernetes.io/projected/6e4d93e2-3440-4e06-ac18-90abf97564c2-kube-api-access-66bc4\") pod \"payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:12.449672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.444191 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dcd066-f5e5-42ef-afa2-b109b5f183dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.549238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.549198 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4d93e2-3440-4e06-ac18-90abf97564c2-plugins-config-volume\") pod \"payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:12.549818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.549788 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bc4\" (UniqueName: \"kubernetes.io/projected/6e4d93e2-3440-4e06-ac18-90abf97564c2-kube-api-access-66bc4\") pod \"payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:12.551290 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.550497 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx82k\" (UniqueName: \"kubernetes.io/projected/85dcd066-f5e5-42ef-afa2-b109b5f183dc-kube-api-access-cx82k\") pod \"payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.551537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.551507 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4d93e2-3440-4e06-ac18-90abf97564c2-plugins-config-volume\") pod \"payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:12.581194 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.581159 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bc4\" (UniqueName: \"kubernetes.io/projected/6e4d93e2-3440-4e06-ac18-90abf97564c2-kube-api-access-66bc4\") pod \"payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:12.750683 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.750624 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:12.864727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:12.864398 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:13.041995 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.041962 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4"] Nov 03 19:47:13.043301 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:47:13.043275 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dcd066_f5e5_42ef_afa2_b109b5f183dc.slice/crio-7f08c4124bac7e8775a769959ec58b242c277ec9e0fd2976b7c8f27bca5dd454 WatchSource:0}: Error finding container 7f08c4124bac7e8775a769959ec58b242c277ec9e0fd2976b7c8f27bca5dd454: Status 404 returned error can't find the container with id 7f08c4124bac7e8775a769959ec58b242c277ec9e0fd2976b7c8f27bca5dd454 Nov 03 19:47:13.199824 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:47:13.199788 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4d93e2_3440_4e06_ac18_90abf97564c2.slice/crio-4c38953509c213f3dd6d429060ea2c4c2316ba88a4b918447f5db193b1370cac WatchSource:0}: Error finding container 4c38953509c213f3dd6d429060ea2c4c2316ba88a4b918447f5db193b1370cac: Status 404 returned error can't find the container with id 4c38953509c213f3dd6d429060ea2c4c2316ba88a4b918447f5db193b1370cac Nov 03 19:47:13.200217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.200006 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6"] Nov 03 19:47:13.621611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.621242 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" event={"ID":"85dcd066-f5e5-42ef-afa2-b109b5f183dc","Type":"ContainerStarted","Data":"c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc"} Nov 03 19:47:13.621611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.621323 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" event={"ID":"85dcd066-f5e5-42ef-afa2-b109b5f183dc","Type":"ContainerStarted","Data":"7f08c4124bac7e8775a769959ec58b242c277ec9e0fd2976b7c8f27bca5dd454"} Nov 03 19:47:13.623599 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.623554 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:13.626640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.626598 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" event={"ID":"6e4d93e2-3440-4e06-ac18-90abf97564c2","Type":"ContainerStarted","Data":"7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63"} Nov 03 19:47:13.626774 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.626674 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" event={"ID":"6e4d93e2-3440-4e06-ac18-90abf97564c2","Type":"ContainerStarted","Data":"4c38953509c213f3dd6d429060ea2c4c2316ba88a4b918447f5db193b1370cac"} Nov 03 19:47:13.626904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.626865 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:13.646146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.646075 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" podStartSLOduration=1.646059154 podStartE2EDuration="1.646059154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:13.645263445 +0000 UTC m=+941.350102799" watchObservedRunningTime="2026-11-03 19:47:13.646059154 +0000 UTC m=+941.350898495" Nov 03 19:47:13.682707 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:13.682640 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" podStartSLOduration=1.682621836 podStartE2EDuration="1.682621836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:13.673868001 +0000 UTC m=+941.378707342" watchObservedRunningTime="2026-11-03 19:47:13.682621836 +0000 UTC m=+941.387461177" Nov 03 19:47:17.321922 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:17.321848 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:22.021848 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:22.021804 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:47:24.668219 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:24.666762 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:25.644852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:25.644818 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:31.319608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.265674 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:31.373739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.373683 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:31.398606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.376661 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:31.398606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.394666 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 03 19:47:31.399021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.398967 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:47:31.399490 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.399461 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:47:31.514725 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.514672 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:31.860792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.860723 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:31.961032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:31.949104 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:47:32.173972 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.173864 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:47:32.274123 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.272680 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:32.292610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.274622 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:47:32.475675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.475557 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:32.498619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.487393 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:32.624189 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.475713 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:33.117425 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.488298 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:47:33.142749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:33.142714 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:47:33.228608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.553887 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:33.228608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:33.217890 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:33.228608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.933623 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:47:33.251606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:32.962771 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:33.356607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:33.275541 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:33.483268 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:33.483167 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:33.569610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:33.559702 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded" Nov 03 19:47:33.630794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:33.630747 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:34.030627 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:34.030575 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:47:34.034734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:34.034709 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:47:34.038602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:34.034949 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:47:34.998743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:34.118909 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:47:35.030604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:34.260073 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:35.030873 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:34.572812 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:47:35.031068 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:34.704783 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95" podUID="3e2993e6-3c87-42aa-9279-fc3d11d9e1c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:35.065161 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.065110 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:35.068362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.068311 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:47:35.116951 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.116913 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:35.117260 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.117209 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:35.117639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.117612 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:35.154605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.117965 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl" podUID="0067d07b-adb9-4976-a4fb-6bdcec9091cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:35.326231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.326132 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:35.451184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.432637 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:47:35.481618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.475444 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 19:47:35.515606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.493197 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 03 19:47:35.827642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:35.816161 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 19:47:36.066615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.039195 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:47:36.094391 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.094283 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:47:36.109616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.100666 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:47:36.234603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.234112 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:47:36.260609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.235951 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:36.334070 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.333801 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:36.379661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.378691 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:36.416606 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:36.412736 4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.324s" Nov 03 19:47:36.416606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.412889 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:47:36.416606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.412921 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng"] Nov 03 19:47:36.451133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.451077 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:36.909637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.900426 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng"] Nov 03 19:47:36.909637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.900610 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:47:36.909637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.906095 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:36.918604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.914901 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 19:47:36.960608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.919707 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 19:47:36.961406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.919764 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 19:47:36.965603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.919917 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-79jcj\"" Nov 03 19:47:37.051333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.973882 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.051333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.973985 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.051333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.974135 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.051333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.974179 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.051333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.974244 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e5d0aa-673f-4bda-b5b0-a9f769389052-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.051333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.974308 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.051333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:36.974555 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6g2\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-kube-api-access-pg6g2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.081507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.075942 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:37.143140 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.143094 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:47:37.167016 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.164092 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:37.219874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.219831 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6g2\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-kube-api-access-pg6g2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.220300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.220266 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.230806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.220483 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.231274 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.231250 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.238714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.238681 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.238970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.238948 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e5d0aa-673f-4bda-b5b0-a9f769389052-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.314422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.314383 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.315038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.278055 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:37.315561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.229942 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.350955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.289419 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:37.406736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.393697 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:37.420728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.409557 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.443617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.422689 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:37.477513 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.451648 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:47:37.542670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.514916 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.605327 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.605293 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e5d0aa-673f-4bda-b5b0-a9f769389052-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.655932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.643180 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.675279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.671341 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6g2\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-kube-api-access-pg6g2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.797943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.785220 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9r5ng\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:37.889606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.859549 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:38.121138 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:37.866903 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:38.142950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.079921 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:38.153616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.147842 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:38.175733 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.175686 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:38.328422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.327723 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:38.352742 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.352701 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:47:38.411661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.410890 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:38.498935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.498883 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:38.504662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:38.504617 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:39.110617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:39.100249 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 19:47:39.161713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:39.145771 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:39.161713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:39.150652 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:47:39.179868 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:39.179817 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:47:39.402848 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:39.368565 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:39.893240 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:39.798608 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:40.162800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.162675 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:40.163420 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.163110 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:47:40.182743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.182693 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:40.289693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.289641 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:47:40.403355 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.397990 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:47:40.710986 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.678360 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:47:40.710986 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.678815 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:47:40.799995 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.799947 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:47:40.800304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.800259 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:40.800525 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:40.800505 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:47:41.116667 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.113625 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-sxc97" podUID="da4973d4-3de9-4f1e-96cb-2def28c1396f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Nov 03 19:47:41.170610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.165788 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:41.222639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.186832 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:41.439599 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.439482 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:41.443672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.192107 4314 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.006477124s: [/var/lib/containers/storage/overlay/7fac382fe56a94b934890c822be16f46c2928b369183ccfee9192dd94ed5efa3/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-bd9746b54-5blw5_e0989ad6-060e-4da2-a646-6095e895ad09/manager/0.log]; will not log again for this container unless duration exceeds 2s Nov 03 19:47:41.443840 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.296651 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:47:41.577618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.523599 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 03 19:47:41.577618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.523686 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:47:41.577618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.191864 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:47:41.683104 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:41.636546 4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.148s" Nov 03 19:47:41.683104 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.636655 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng"] Nov 03 19:47:41.683104 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.636708 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp"] Nov 03 19:47:41.863912 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.863867 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:47:41.890729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.890694 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:47:41.899611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.899566 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:47:41.938314 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.938283 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:47:41.944454 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.944429 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:47:41.981476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:41.981437 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" event={"ID":"82e5d0aa-673f-4bda-b5b0-a9f769389052","Type":"ContainerStarted","Data":"a43a7b1acf385fb037cdd584fbf8dc3ceab658ba9ae9e17e21b7f2cde8fba235"} Nov 03 19:47:42.038458 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:42.038433 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:47:42.039496 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:42.039478 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:47:42.043177 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:42.043155 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:47:43.215644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.215606 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn"] Nov 03 19:47:43.216279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.216242 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerName="istio-proxy" containerID="cri-o://dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d" gracePeriod=30 Nov 03 19:47:43.320216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.320177 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb"] Nov 03 19:47:43.327530 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.327505 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb"] Nov 03 19:47:43.327979 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.327851 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.344616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.343575 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2e17271b-openshift-default-dockercfg-pdbqv\"" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.423616 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b8828-5b16-4416-96d2-59df45e48994-istio-podinfo\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.423662 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-istio-token\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.423809 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.424406 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b8828-5b16-4416-96d2-59df45e48994-istiod-ca-cert\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.424547 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvczt\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-kube-api-access-bvczt\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.424593 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-credential-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.424632 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-data\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.424697 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-envoy\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.428653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.424723 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-certs\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.538520 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b8828-5b16-4416-96d2-59df45e48994-istio-podinfo\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.538577 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-istio-token\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.538783 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.538877 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b8828-5b16-4416-96d2-59df45e48994-istiod-ca-cert\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.539037 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvczt\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-kube-api-access-bvczt\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.539076 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-credential-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.539110 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-data\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.539188 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-envoy\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.540623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.539232 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-certs\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.573636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.573601 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.618635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.584981 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-credential-socket\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.618635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.614976 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-certs\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.619711 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.619572 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-data\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.629678 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.625777 4314 patch_prober.go:29] interesting pod/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:47:43.629678 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.625830 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:47:43.716275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.653968 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b8828-5b16-4416-96d2-59df45e48994-istiod-ca-cert\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.716275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.713840 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-envoy\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.716275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.716196 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvczt\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-kube-api-access-bvczt\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.721810 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.721777 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-istio-token\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.722013 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.721929 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b8828-5b16-4416-96d2-59df45e48994-istio-podinfo\") pod \"e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:43.917986 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:43.917892 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:44.295980 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:44.295066 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb"] Nov 03 19:47:44.308448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:44.308411 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:44.308565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:44.308481 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:44.308565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:44.308511 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:45.034528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:45.034481 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" event={"ID":"ab1b8828-5b16-4416-96d2-59df45e48994","Type":"ContainerStarted","Data":"b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105"} Nov 03 19:47:45.034819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:45.034553 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" event={"ID":"ab1b8828-5b16-4416-96d2-59df45e48994","Type":"ContainerStarted","Data":"d8adb9c636044486ccef571768920b14dc8b1f48a67309205142d504e664746e"} Nov 03 19:47:45.063555 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:45.063492 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" podStartSLOduration=2.063477058 podStartE2EDuration="2.063477058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:45.057906486 +0000 UTC m=+972.762745838" watchObservedRunningTime="2026-11-03 19:47:45.063477058 +0000 UTC m=+972.768316395" Nov 03 19:47:45.920600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:45.918220 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:45.920600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:45.919215 4314 patch_prober.go:29] interesting pod/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:47:45.920600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:45.919256 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 19:47:46.514604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:46.513862 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb"] Nov 03 19:47:46.637388 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:46.637333 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" podUID="7929582c-f39d-4d78-b8f8-2342bbefeb29" containerName="payload-pre-processing" containerID="cri-o://b3155e546d7dd06095f9f7bd8d92d75b6892cbfe9505a39d98f9dbd988fa4c14" gracePeriod=25 Nov 03 19:47:46.929610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:46.927836 4314 patch_prober.go:29] interesting pod/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:47:46.929610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:46.927899 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 19:47:47.111607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.111423 4314 generic.go:356] "Generic (PLEG): container finished" podID="7929582c-f39d-4d78-b8f8-2342bbefeb29" containerID="b3155e546d7dd06095f9f7bd8d92d75b6892cbfe9505a39d98f9dbd988fa4c14" exitCode=0 Nov 03 19:47:47.111806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.111619 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" event={"ID":"7929582c-f39d-4d78-b8f8-2342bbefeb29","Type":"ContainerDied","Data":"b3155e546d7dd06095f9f7bd8d92d75b6892cbfe9505a39d98f9dbd988fa4c14"} Nov 03 19:47:47.125148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.125106 4314 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" secret="" err="secret \"e2e-ait-2e17271b-openshift-default-dockercfg-pdbqv\" not found" Nov 03 19:47:47.441321 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.440771 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:47.631975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.631927 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7929582c-f39d-4d78-b8f8-2342bbefeb29-kube-api-access-rbksm\" (UniqueName: \"kubernetes.io/projected/7929582c-f39d-4d78-b8f8-2342bbefeb29-kube-api-access-rbksm\") pod \"7929582c-f39d-4d78-b8f8-2342bbefeb29\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " Nov 03 19:47:47.632417 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.632385 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7929582c-f39d-4d78-b8f8-2342bbefeb29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7929582c-f39d-4d78-b8f8-2342bbefeb29-plugins-config-volume\") pod \"7929582c-f39d-4d78-b8f8-2342bbefeb29\" (UID: \"7929582c-f39d-4d78-b8f8-2342bbefeb29\") " Nov 03 19:47:47.642653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.635617 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7929582c-f39d-4d78-b8f8-2342bbefeb29-plugins-config-volume" pod "7929582c-f39d-4d78-b8f8-2342bbefeb29" (UID: "7929582c-f39d-4d78-b8f8-2342bbefeb29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:47.718665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.717379 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7929582c-f39d-4d78-b8f8-2342bbefeb29-kube-api-access-rbksm" pod "7929582c-f39d-4d78-b8f8-2342bbefeb29" (UID: "7929582c-f39d-4d78-b8f8-2342bbefeb29"). InnerVolumeSpecName "kube-api-access-rbksm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:47.737787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.737750 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7929582c-f39d-4d78-b8f8-2342bbefeb29-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:47.737787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.737786 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbksm\" (UniqueName: \"kubernetes.io/projected/7929582c-f39d-4d78-b8f8-2342bbefeb29-kube-api-access-rbksm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:47.951653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.938009 4314 patch_prober.go:29] interesting pod/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 19:47:47.951653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:47.938075 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 19:47:48.146607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.146419 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" containerName="istio-proxy" containerID="cri-o://b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105" gracePeriod=30 Nov 03 19:47:48.146607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.146574 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" Nov 03 19:47:48.153928 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.153172 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" event={"ID":"7929582c-f39d-4d78-b8f8-2342bbefeb29","Type":"ContainerDied","Data":"33fc1b19da4d0acfd2a10fffa46ef3007305eb18c4437839bfe1eedccf0303c5"} Nov 03 19:47:48.153928 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.153314 4314 scope.go:122] "RemoveContainer" containerID="b3155e546d7dd06095f9f7bd8d92d75b6892cbfe9505a39d98f9dbd988fa4c14" Nov 03 19:47:48.226915 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.226859 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp"] Nov 03 19:47:48.231618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.231589 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp"] Nov 03 19:47:48.321057 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:47:48.321025 4314 logging.go:55] [core] [Channel #731 SubChannel #732]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.87:50052", ServerName: "10.134.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.87:50052: operation was canceled" Nov 03 19:47:48.321625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.321164 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-cbfcf47a-65cb5b8c5f-7jmsp" podUID="7929582c-f39d-4d78-b8f8-2342bbefeb29" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.87:50052\" within 1s: context deadline exceeded" Nov 03 19:47:48.435660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:48.435549 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7929582c-f39d-4d78-b8f8-2342bbefeb29" path="/var/lib/kubelet/pods/7929582c-f39d-4d78-b8f8-2342bbefeb29/volumes" Nov 03 19:47:50.667669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:50.667408 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:50.667669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:50.667502 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:47:51.623624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:51.620798 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" event={"ID":"82e5d0aa-673f-4bda-b5b0-a9f769389052","Type":"ContainerStarted","Data":"edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75"} Nov 03 19:47:51.624951 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:51.624724 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:51.626251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:51.625969 4314 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9r5ng container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:47:51.626251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:51.626023 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" podUID="82e5d0aa-673f-4bda-b5b0-a9f769389052" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:47:51.658070 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:51.658004 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" podStartSLOduration=8.607364283 podStartE2EDuration="18.657984922s" totalImagesPullingTime="10.050620639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:51.655193346 +0000 UTC m=+979.360032700" watchObservedRunningTime="2026-11-03 19:47:51.657984922 +0000 UTC m=+979.362824265" Nov 03 19:47:52.014107 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:52.013323 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6"] Nov 03 19:47:52.054773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:52.050064 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4"] Nov 03 19:47:52.081151 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:52.081069 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:47:52.669206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:52.668734 4314 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9r5ng container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:47:52.669206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:52.668810 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" podUID="82e5d0aa-673f-4bda-b5b0-a9f769389052" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:47:53.668849 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:53.668819 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 19:47:54.302430 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.302407 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:47:54.366192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366159 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-data\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366347 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366264 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-socket\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366347 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366298 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-token\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-token\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366465 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366442 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-kube-api-access-s8g4s\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-kube-api-access-s8g4s\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366492 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-envoy\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366602 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a093b1d-8e0e-4703-8aac-149aae6db744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a093b1d-8e0e-4703-8aac-149aae6db744-istiod-ca-cert\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366640 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-podinfo\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366669 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-credential-socket\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.366846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366668 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-socket" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:54.366846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366740 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-certs\") pod \"6a093b1d-8e0e-4703-8aac-149aae6db744\" (UID: \"6a093b1d-8e0e-4703-8aac-149aae6db744\") " Nov 03 19:47:54.367015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366904 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-data" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:54.367015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.366993 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a093b1d-8e0e-4703-8aac-149aae6db744-istiod-ca-cert" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:54.367273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.367251 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-credential-socket" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:54.367353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.367274 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-certs" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:54.367784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.367692 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.367784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.367718 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.367784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.367733 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a093b1d-8e0e-4703-8aac-149aae6db744-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.367784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.367748 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.367784 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.367762 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.369003 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.368975 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-token" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:54.369772 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.369738 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-podinfo" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:47:54.371557 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.371533 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-envoy" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:54.371887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.371867 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-kube-api-access-s8g4s" pod "6a093b1d-8e0e-4703-8aac-149aae6db744" (UID: "6a093b1d-8e0e-4703-8aac-149aae6db744"). InnerVolumeSpecName "kube-api-access-s8g4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:54.469520 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.469492 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8g4s\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-kube-api-access-s8g4s\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.469520 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.469520 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.469722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.469531 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.469722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.469541 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a093b1d-8e0e-4703-8aac-149aae6db744-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:54.678873 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.678769 4314 generic.go:356] "Generic (PLEG): container finished" podID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerID="dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d" exitCode=0 Nov 03 19:47:54.679724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.679693 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" event={"ID":"6a093b1d-8e0e-4703-8aac-149aae6db744","Type":"ContainerDied","Data":"dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d"} Nov 03 19:47:54.679867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.679757 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" event={"ID":"6a093b1d-8e0e-4703-8aac-149aae6db744","Type":"ContainerDied","Data":"5708b91c77491a14234f133c4d9254e7d132883cec0460e0cafdbffe3c96d7b9"} Nov 03 19:47:54.679867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.679787 4314 scope.go:122] "RemoveContainer" containerID="dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d" Nov 03 19:47:54.679981 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.679699 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn" Nov 03 19:47:54.706122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.706098 4314 scope.go:122] "RemoveContainer" containerID="dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d" Nov 03 19:47:54.706490 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:54.706463 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d\": container with ID starting with dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d not found: ID does not exist" containerID="dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d" Nov 03 19:47:54.706572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.706507 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d"} err="failed to get container status \"dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d\": rpc error: code = NotFound desc = could not find container \"dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d\": container with ID starting with dc1d66b7150fb80d7eee34f3e63f4c829ec7ac1b6a098503b64fa4aaeb78711d not found: ID does not exist" Nov 03 19:47:54.717956 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.717926 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn"] Nov 03 19:47:54.722042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:54.722021 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cbfcf47a-openshift-default-659f56fb9-dgfvn"] Nov 03 19:47:56.430216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:56.430184 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" path="/var/lib/kubelet/pods/6a093b1d-8e0e-4703-8aac-149aae6db744/volumes" Nov 03 19:47:57.014519 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.014475 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" podUID="6e4d93e2-3440-4e06-ac18-90abf97564c2" containerName="payload-processing" containerID="cri-o://7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63" gracePeriod=25 Nov 03 19:47:57.051442 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.051401 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" podUID="85dcd066-f5e5-42ef-afa2-b109b5f183dc" containerName="payload-pre-processing" containerID="cri-o://c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc" gracePeriod=25 Nov 03 19:47:57.423086 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.422763 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:57.512862 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.512831 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:57.516986 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.516965 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:57.528088 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.528064 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4d93e2-3440-4e06-ac18-90abf97564c2-kube-api-access-66bc4\" (UniqueName: \"kubernetes.io/projected/6e4d93e2-3440-4e06-ac18-90abf97564c2-kube-api-access-66bc4\") pod \"6e4d93e2-3440-4e06-ac18-90abf97564c2\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " Nov 03 19:47:57.528244 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.528120 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4d93e2-3440-4e06-ac18-90abf97564c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4d93e2-3440-4e06-ac18-90abf97564c2-plugins-config-volume\") pod \"6e4d93e2-3440-4e06-ac18-90abf97564c2\" (UID: \"6e4d93e2-3440-4e06-ac18-90abf97564c2\") " Nov 03 19:47:57.528564 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.528536 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4d93e2-3440-4e06-ac18-90abf97564c2-plugins-config-volume" pod "6e4d93e2-3440-4e06-ac18-90abf97564c2" (UID: "6e4d93e2-3440-4e06-ac18-90abf97564c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:57.529485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.529464 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4d93e2-3440-4e06-ac18-90abf97564c2-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.530536 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.530510 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4d93e2-3440-4e06-ac18-90abf97564c2-kube-api-access-66bc4" pod "6e4d93e2-3440-4e06-ac18-90abf97564c2" (UID: "6e4d93e2-3440-4e06-ac18-90abf97564c2"). InnerVolumeSpecName "kube-api-access-66bc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:57.631150 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631055 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-istio-token\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631184 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-socket\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631209 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-credential-socket\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631493 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631409 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-certs\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631493 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631456 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85dcd066-f5e5-42ef-afa2-b109b5f183dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dcd066-f5e5-42ef-afa2-b109b5f183dc-plugins-config-volume\") pod \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " Nov 03 19:47:57.631493 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631483 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-data\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631536 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-credential-socket" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:57.631710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631556 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dcd066-f5e5-42ef-afa2-b109b5f183dc-kube-api-access-cx82k\" (UniqueName: \"kubernetes.io/projected/85dcd066-f5e5-42ef-afa2-b109b5f183dc-kube-api-access-cx82k\") pod \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\" (UID: \"85dcd066-f5e5-42ef-afa2-b109b5f183dc\") " Nov 03 19:47:57.631710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631553 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-socket" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:57.631710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631630 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab1b8828-5b16-4416-96d2-59df45e48994-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b8828-5b16-4416-96d2-59df45e48994-istio-podinfo\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631681 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-certs" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:57.631955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631721 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-envoy\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631770 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-kube-api-access-bvczt\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-kube-api-access-bvczt\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.631955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631899 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1b8828-5b16-4416-96d2-59df45e48994-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b8828-5b16-4416-96d2-59df45e48994-istiod-ca-cert\") pod \"ab1b8828-5b16-4416-96d2-59df45e48994\" (UID: \"ab1b8828-5b16-4416-96d2-59df45e48994\") " Nov 03 19:47:57.632210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631772 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-data" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:57.632268 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.631943 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dcd066-f5e5-42ef-afa2-b109b5f183dc-plugins-config-volume" pod "85dcd066-f5e5-42ef-afa2-b109b5f183dc" (UID: "85dcd066-f5e5-42ef-afa2-b109b5f183dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:57.632788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.632757 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1b8828-5b16-4416-96d2-59df45e48994-istiod-ca-cert" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:47:57.632918 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.632891 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66bc4\" (UniqueName: \"kubernetes.io/projected/6e4d93e2-3440-4e06-ac18-90abf97564c2-kube-api-access-66bc4\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.632918 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.632922 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.633103 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.632938 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dcd066-f5e5-42ef-afa2-b109b5f183dc-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.633103 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.632952 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.633103 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.632964 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.633103 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.632979 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.634154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.634125 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-istio-token" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:57.634154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.634148 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab1b8828-5b16-4416-96d2-59df45e48994-istio-podinfo" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:47:57.634359 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.634166 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dcd066-f5e5-42ef-afa2-b109b5f183dc-kube-api-access-cx82k" pod "85dcd066-f5e5-42ef-afa2-b109b5f183dc" (UID: "85dcd066-f5e5-42ef-afa2-b109b5f183dc"). InnerVolumeSpecName "kube-api-access-cx82k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:57.634499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.634482 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-envoy" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:47:57.634563 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.634536 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-kube-api-access-bvczt" pod "ab1b8828-5b16-4416-96d2-59df45e48994" (UID: "ab1b8828-5b16-4416-96d2-59df45e48994"). InnerVolumeSpecName "kube-api-access-bvczt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:47:57.718851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.718536 4314 generic.go:356] "Generic (PLEG): container finished" podID="85dcd066-f5e5-42ef-afa2-b109b5f183dc" containerID="c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc" exitCode=0 Nov 03 19:47:57.718851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.718691 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" Nov 03 19:47:57.718851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.718694 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" event={"ID":"85dcd066-f5e5-42ef-afa2-b109b5f183dc","Type":"ContainerDied","Data":"c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc"} Nov 03 19:47:57.718851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.718748 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4" event={"ID":"85dcd066-f5e5-42ef-afa2-b109b5f183dc","Type":"ContainerDied","Data":"7f08c4124bac7e8775a769959ec58b242c277ec9e0fd2976b7c8f27bca5dd454"} Nov 03 19:47:57.718851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.718777 4314 scope.go:122] "RemoveContainer" containerID="c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc" Nov 03 19:47:57.721526 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.721485 4314 generic.go:356] "Generic (PLEG): container finished" podID="6e4d93e2-3440-4e06-ac18-90abf97564c2" containerID="7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63" exitCode=0 Nov 03 19:47:57.721700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.721676 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" event={"ID":"6e4d93e2-3440-4e06-ac18-90abf97564c2","Type":"ContainerDied","Data":"7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63"} Nov 03 19:47:57.721764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.721722 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" event={"ID":"6e4d93e2-3440-4e06-ac18-90abf97564c2","Type":"ContainerDied","Data":"4c38953509c213f3dd6d429060ea2c4c2316ba88a4b918447f5db193b1370cac"} Nov 03 19:47:57.721824 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.721799 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6" Nov 03 19:47:57.724630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.724558 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" Nov 03 19:47:57.725192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.725004 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" event={"ID":"ab1b8828-5b16-4416-96d2-59df45e48994","Type":"ContainerDied","Data":"b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105"} Nov 03 19:47:57.725192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.724516 4314 generic.go:356] "Generic (PLEG): container finished" podID="ab1b8828-5b16-4416-96d2-59df45e48994" containerID="b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105" exitCode=0 Nov 03 19:47:57.725192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.725114 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb" event={"ID":"ab1b8828-5b16-4416-96d2-59df45e48994","Type":"ContainerDied","Data":"d8adb9c636044486ccef571768920b14dc8b1f48a67309205142d504e664746e"} Nov 03 19:47:57.737970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.737941 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.738127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.737975 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx82k\" (UniqueName: \"kubernetes.io/projected/85dcd066-f5e5-42ef-afa2-b109b5f183dc-kube-api-access-cx82k\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.738127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.737993 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1b8828-5b16-4416-96d2-59df45e48994-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.738127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.738006 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1b8828-5b16-4416-96d2-59df45e48994-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.738127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.738020 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvczt\" (UniqueName: \"kubernetes.io/projected/ab1b8828-5b16-4416-96d2-59df45e48994-kube-api-access-bvczt\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.738127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.738033 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1b8828-5b16-4416-96d2-59df45e48994-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:47:57.741988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.741954 4314 scope.go:122] "RemoveContainer" containerID="c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc" Nov 03 19:47:57.742376 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:57.742350 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc\": container with ID starting with c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc not found: ID does not exist" containerID="c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc" Nov 03 19:47:57.742475 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.742382 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc"} err="failed to get container status \"c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc\": rpc error: code = NotFound desc = could not find container \"c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc\": container with ID starting with c596158bf496f6763c5e1eeda22f6a70e2b425ac7794baa28e9cb21787546ffc not found: ID does not exist" Nov 03 19:47:57.742475 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.742399 4314 scope.go:122] "RemoveContainer" containerID="7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63" Nov 03 19:47:57.766569 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.766547 4314 scope.go:122] "RemoveContainer" containerID="7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63" Nov 03 19:47:57.768805 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:57.766961 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63\": container with ID starting with 7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63 not found: ID does not exist" containerID="7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63" Nov 03 19:47:57.768805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.766997 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63"} err="failed to get container status \"7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63\": rpc error: code = NotFound desc = could not find container \"7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63\": container with ID starting with 7cc91418d19ed8101fe2ad44efdbb2acfeaf3ebe1cf04e4b9b54af89d8312a63 not found: ID does not exist" Nov 03 19:47:57.768805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.767023 4314 scope.go:122] "RemoveContainer" containerID="b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105" Nov 03 19:47:57.782332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.782311 4314 scope.go:122] "RemoveContainer" containerID="b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105" Nov 03 19:47:57.782754 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:47:57.782699 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105\": container with ID starting with b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105 not found: ID does not exist" containerID="b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105" Nov 03 19:47:57.782861 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.782766 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105"} err="failed to get container status \"b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105\": rpc error: code = NotFound desc = could not find container \"b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105\": container with ID starting with b6cfe31f415bbcb9c8f5c76cd530f14da4b40066abafe49aa1b4f79aecef7105 not found: ID does not exist" Nov 03 19:47:57.792359 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.792338 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4"] Nov 03 19:47:57.799359 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.799161 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-26c00c22-8ffd65f95-tcdh4"] Nov 03 19:47:57.806717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.806688 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb"] Nov 03 19:47:57.814737 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.814678 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2e17271b-openshift-default-57bc7cb65f-gvnnb"] Nov 03 19:47:57.820619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.820553 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6"] Nov 03 19:47:57.826249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:57.826226 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-26c00c22-78cf5b6bf5-gcdw6"] Nov 03 19:47:58.487406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:58.487321 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4d93e2-3440-4e06-ac18-90abf97564c2" path="/var/lib/kubelet/pods/6e4d93e2-3440-4e06-ac18-90abf97564c2/volumes" Nov 03 19:47:58.487968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:58.487946 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dcd066-f5e5-42ef-afa2-b109b5f183dc" path="/var/lib/kubelet/pods/85dcd066-f5e5-42ef-afa2-b109b5f183dc/volumes" Nov 03 19:47:58.488590 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:47:58.488565 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" path="/var/lib/kubelet/pods/ab1b8828-5b16-4416-96d2-59df45e48994/volumes" Nov 03 19:48:00.516616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:00.514154 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk"] Nov 03 19:48:00.516616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:00.514424 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" podUID="9bba2368-e9a4-4ea6-8787-3e3711f5da2f" containerName="istio-proxy" containerID="cri-o://9d9932792c954ce2e4cb7522336fbe9c3d778f34aaacccd34cf28b11710e7dea" gracePeriod=30 Nov 03 19:48:05.832906 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.832837 4314 generic.go:356] "Generic (PLEG): container finished" podID="9bba2368-e9a4-4ea6-8787-3e3711f5da2f" containerID="9d9932792c954ce2e4cb7522336fbe9c3d778f34aaacccd34cf28b11710e7dea" exitCode=0 Nov 03 19:48:05.834105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.833049 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" event={"ID":"9bba2368-e9a4-4ea6-8787-3e3711f5da2f","Type":"ContainerDied","Data":"9d9932792c954ce2e4cb7522336fbe9c3d778f34aaacccd34cf28b11710e7dea"} Nov 03 19:48:05.834105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.833133 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" event={"ID":"9bba2368-e9a4-4ea6-8787-3e3711f5da2f","Type":"ContainerDied","Data":"f2a95608f9bfbe0e2b06da110d2022a985507fc94bf38f77607aa9629af9211f"} Nov 03 19:48:05.834105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.833181 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2a95608f9bfbe0e2b06da110d2022a985507fc94bf38f77607aa9629af9211f" Nov 03 19:48:05.861636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.857512 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:48:05.964644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.964613 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-credential-socket\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.964943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.964924 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-envoy\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.965284 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.965250 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-credential-socket" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:48:05.965486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.965423 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istiod-ca-cert\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.965779 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.965759 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-podinfo\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.965931 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.965914 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-certs\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.966107 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.966090 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-kube-api-access-5fmtx\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-kube-api-access-5fmtx\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.966326 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.966309 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-data\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.966598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.966555 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-token\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-token\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.966729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.966179 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istiod-ca-cert" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:48:05.966861 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.966843 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-socket\") pod \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\" (UID: \"9bba2368-e9a4-4ea6-8787-3e3711f5da2f\") " Nov 03 19:48:05.967251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.966873 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-certs" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:48:05.967387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.967118 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-socket" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:48:05.967917 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.967893 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:05.968033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.968020 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:05.968119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.968108 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:05.968254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.968240 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:05.968410 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.968390 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-data" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:48:05.969077 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.969055 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-envoy" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:48:05.969461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.969440 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-kube-api-access-5fmtx" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "kube-api-access-5fmtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:48:05.969622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.969505 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-token" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:48:05.970008 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:05.969984 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-podinfo" pod "9bba2368-e9a4-4ea6-8787-3e3711f5da2f" (UID: "9bba2368-e9a4-4ea6-8787-3e3711f5da2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:48:06.070613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.070509 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:06.070613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.070547 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:06.070613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.070563 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fmtx\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-kube-api-access-5fmtx\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:06.070613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.070594 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:06.070613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.070608 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba2368-e9a4-4ea6-8787-3e3711f5da2f-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:48:06.839537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.839500 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk" Nov 03 19:48:06.862136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.862073 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk"] Nov 03 19:48:06.866202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:06.866174 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-26c00c22-openshift-default-7cdbc5f7d9-42ghk"] Nov 03 19:48:08.429385 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:08.429353 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bba2368-e9a4-4ea6-8787-3e3711f5da2f" path="/var/lib/kubelet/pods/9bba2368-e9a4-4ea6-8787-3e3711f5da2f/volumes" Nov 03 19:48:09.758341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:09.758309 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:48:12.630831 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:12.630790 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:48:12.631439 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:12.631404 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:13.072607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.070689 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:13.092195 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.092141 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:48:13.133218 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.133176 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:13.155804 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.155754 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:13.412662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.411430 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:13.481606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.480683 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:13.552626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.536692 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:48:13.931985 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:13.930783 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.079605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.079467 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.195344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.093556 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.195344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.129690 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:48:14.195344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.130085 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:48:14.195344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.147202 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.195344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.149239 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:48:14.195344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.156883 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.195344 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.157935 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:48:14.331608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.331495 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:48:14.338616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.332762 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.476660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.476336 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.676608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.635740 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.900634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.875686 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.917094 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.906763 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:14.976971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:14.976918 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:15.169852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.169745 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:15.176096 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.176058 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:15.181882 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.181840 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:48:15.220945 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.220912 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:48:15.221205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.221190 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:48:15.221349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.221337 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:48:15.335292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.335248 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:48:15.336962 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.336921 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:48:15.356145 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.351016 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:48:15.356145 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.352433 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:48:15.455701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.454020 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:48:15.463887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.463814 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:48:15.533961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:15.533929 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:48:21.998097 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:48:21.998055 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:48:30.953126 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.953090 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n"] Nov 03 19:48:30.953972 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.953944 4314 cpu_manager.go:414] "containerMap: removing container" podUID="9bba2368-e9a4-4ea6-8787-3e3711f5da2f" containerName="istio-proxy" Nov 03 19:48:30.953972 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.953971 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bba2368-e9a4-4ea6-8787-3e3711f5da2f" containerName="istio-proxy" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954002 4314 cpu_manager.go:414] "containerMap: removing container" podUID="6e4d93e2-3440-4e06-ac18-90abf97564c2" containerName="payload-processing" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954009 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4d93e2-3440-4e06-ac18-90abf97564c2" containerName="payload-processing" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954031 4314 cpu_manager.go:414] "containerMap: removing container" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerName="istio-proxy" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954039 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerName="istio-proxy" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954065 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7929582c-f39d-4d78-b8f8-2342bbefeb29" containerName="payload-pre-processing" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954074 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7929582c-f39d-4d78-b8f8-2342bbefeb29" containerName="payload-pre-processing" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954091 4314 cpu_manager.go:414] "containerMap: removing container" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" containerName="istio-proxy" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954098 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" containerName="istio-proxy" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954138 4314 cpu_manager.go:414] "containerMap: removing container" podUID="85dcd066-f5e5-42ef-afa2-b109b5f183dc" containerName="payload-pre-processing" Nov 03 19:48:30.954249 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954146 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dcd066-f5e5-42ef-afa2-b109b5f183dc" containerName="payload-pre-processing" Nov 03 19:48:30.955392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954292 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4d93e2-3440-4e06-ac18-90abf97564c2" containerName="payload-processing" Nov 03 19:48:30.955392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954312 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a093b1d-8e0e-4703-8aac-149aae6db744" containerName="istio-proxy" Nov 03 19:48:30.955392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954326 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7929582c-f39d-4d78-b8f8-2342bbefeb29" containerName="payload-pre-processing" Nov 03 19:48:30.955392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954338 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1b8828-5b16-4416-96d2-59df45e48994" containerName="istio-proxy" Nov 03 19:48:30.955392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954351 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dcd066-f5e5-42ef-afa2-b109b5f183dc" containerName="payload-pre-processing" Nov 03 19:48:30.955392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.954363 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bba2368-e9a4-4ea6-8787-3e3711f5da2f" containerName="istio-proxy" Nov 03 19:48:30.960459 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.960435 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:30.961640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:30.961616 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n"] Nov 03 19:48:31.052603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.022639 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7c86d7-e349-4408-be55-626b637e7db6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:31.093644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.093600 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8z25\" (UniqueName: \"kubernetes.io/projected/ad7c86d7-e349-4408-be55-626b637e7db6-kube-api-access-g8z25\") pod \"payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:31.120607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.027899 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-6eb768-dockercfg-vx2jf\"" Nov 03 19:48:31.121222 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.027980 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-6eb768\"" Nov 03 19:48:31.196909 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.196866 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7c86d7-e349-4408-be55-626b637e7db6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:31.198816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.198770 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8z25\" (UniqueName: \"kubernetes.io/projected/ad7c86d7-e349-4408-be55-626b637e7db6-kube-api-access-g8z25\") pod \"payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:31.199658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.197963 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7c86d7-e349-4408-be55-626b637e7db6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:31.208017 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.207954 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8z25\" (UniqueName: \"kubernetes.io/projected/ad7c86d7-e349-4408-be55-626b637e7db6-kube-api-access-g8z25\") pod \"payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:31.280151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.278893 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:31.617152 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.617118 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n"] Nov 03 19:48:31.617504 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:48:31.617480 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7c86d7_e349_4408_be55_626b637e7db6.slice/crio-0e5dafe2b4cebdd850e6e424736d6d3574ad354765dd22bcf7b4a98b0da20517 WatchSource:0}: Error finding container 0e5dafe2b4cebdd850e6e424736d6d3574ad354765dd22bcf7b4a98b0da20517: Status 404 returned error can't find the container with id 0e5dafe2b4cebdd850e6e424736d6d3574ad354765dd22bcf7b4a98b0da20517 Nov 03 19:48:31.679622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:31.679565 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" event={"ID":"ad7c86d7-e349-4408-be55-626b637e7db6","Type":"ContainerStarted","Data":"0e5dafe2b4cebdd850e6e424736d6d3574ad354765dd22bcf7b4a98b0da20517"} Nov 03 19:48:32.696336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:32.696296 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" event={"ID":"ad7c86d7-e349-4408-be55-626b637e7db6","Type":"ContainerStarted","Data":"c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa"} Nov 03 19:48:32.696841 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:32.696815 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:32.714433 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:32.714380 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" podStartSLOduration=2.714363964 podStartE2EDuration="2.714363964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:32.711908893 +0000 UTC m=+1020.416748235" watchObservedRunningTime="2026-11-03 19:48:32.714363964 +0000 UTC m=+1020.419203305" Nov 03 19:48:43.713973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:43.713943 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 19:48:52.048346 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:48:52.048267 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:48:55.538801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.538769 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b"] Nov 03 19:48:55.543790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.543767 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.551015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.550993 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-b9cbea-openshift-default-dockercfg-6lfct\"" Nov 03 19:48:55.566653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.559430 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b"] Nov 03 19:48:55.638321 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638281 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-credential-socket\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.638486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638333 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-istio-token\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.638486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638413 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-data\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.638566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638515 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-socket\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.638700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638677 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrg9\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-kube-api-access-xgrg9\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.638809 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638794 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e508fa96-3607-45d5-ad64-e806599da0a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.638851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638837 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-certs\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.638973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638955 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e508fa96-3607-45d5-ad64-e806599da0a7-istio-podinfo\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.639036 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.638993 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-envoy\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.740932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.740843 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-certs\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.741270 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.741236 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e508fa96-3607-45d5-ad64-e806599da0a7-istio-podinfo\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.741381 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.741291 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-envoy\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.741503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.741486 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-credential-socket\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.741566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.741528 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-istio-token\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.741688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.741560 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-data\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.741688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.741610 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-socket\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.741819 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.741780 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrg9\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-kube-api-access-xgrg9\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.742475 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.742243 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-credential-socket\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.742737 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.742712 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e508fa96-3607-45d5-ad64-e806599da0a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.743065 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.742905 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-data\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.743421 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.743398 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-socket\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.745509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.745486 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e508fa96-3607-45d5-ad64-e806599da0a7-istiod-ca-cert\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.746034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.746013 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-certs\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.749423 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.749396 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-envoy\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.749524 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.749402 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e508fa96-3607-45d5-ad64-e806599da0a7-istio-podinfo\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.757890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.755470 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrg9\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-kube-api-access-xgrg9\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.757890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.756036 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-istio-token\") pod \"e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:55.872650 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:55.872195 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:56.268964 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:56.268922 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b"] Nov 03 19:48:56.269443 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:48:56.269417 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode508fa96_3607_45d5_ad64_e806599da0a7.slice/crio-5b0120a68a84a3f54723046a64f333772b0b671f6ff3702563b347a89ae48a1b WatchSource:0}: Error finding container 5b0120a68a84a3f54723046a64f333772b0b671f6ff3702563b347a89ae48a1b: Status 404 returned error can't find the container with id 5b0120a68a84a3f54723046a64f333772b0b671f6ff3702563b347a89ae48a1b Nov 03 19:48:56.271998 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:56.271960 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:48:56.272113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:56.272034 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:48:56.272113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:56.272066 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:48:57.084775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:57.084671 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" event={"ID":"e508fa96-3607-45d5-ad64-e806599da0a7","Type":"ContainerStarted","Data":"536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484"} Nov 03 19:48:57.084775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:57.084733 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" event={"ID":"e508fa96-3607-45d5-ad64-e806599da0a7","Type":"ContainerStarted","Data":"5b0120a68a84a3f54723046a64f333772b0b671f6ff3702563b347a89ae48a1b"} Nov 03 19:48:57.113928 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:57.113868 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podStartSLOduration=2.113847444 podStartE2EDuration="2.113847444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:57.105623956 +0000 UTC m=+1044.810463296" watchObservedRunningTime="2026-11-03 19:48:57.113847444 +0000 UTC m=+1044.818686787" Nov 03 19:48:57.872778 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:57.872744 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:57.878561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:57.878537 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:58.092760 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:58.092728 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:48:58.093738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:48:58.093712 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:49:06.084609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.083267 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6"] Nov 03 19:49:06.091711 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.089092 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.101128 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.101088 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6"] Nov 03 19:49:06.120871 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.120841 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-2d78ea-openshift-default-dockercfg-9mfjj\"" Nov 03 19:49:06.248003 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.247967 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-certs\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248120 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-podinfo\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248172 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064de3ea-3b4f-483d-8468-62da6e0a19b7-istiod-ca-cert\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248325 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248287 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4lw\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-kube-api-access-sx4lw\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248379 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248342 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-envoy\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248379 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248364 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-socket\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248480 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248451 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-token\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248486 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-data\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.248630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.248575 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-credential-socket\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.352666 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-envoy\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.352716 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-socket\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.352792 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-token\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.352826 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-data\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.352903 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-credential-socket\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.352965 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-certs\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.353086 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-podinfo\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.353125 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064de3ea-3b4f-483d-8468-62da6e0a19b7-istiod-ca-cert\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.353662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.353268 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4lw\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-kube-api-access-sx4lw\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.365608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.354607 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-socket\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.371604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.367909 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-certs\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.371604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.368363 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-data\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.386603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.375179 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4lw\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-kube-api-access-sx4lw\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.386603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.376055 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-credential-socket\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.411605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.389196 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064de3ea-3b4f-483d-8468-62da6e0a19b7-istiod-ca-cert\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.411605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.391888 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-podinfo\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.411605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.392912 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-envoy\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.432603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.428206 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-token\") pod \"e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:06.724229 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:06.724138 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:07.046946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.045853 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6"] Nov 03 19:49:07.054116 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:49:07.054084 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064de3ea_3b4f_483d_8468_62da6e0a19b7.slice/crio-79dae26d840939817a203a3ddb0f98f895f4201007f9bb4e3bc9e54a0b0e213f WatchSource:0}: Error finding container 79dae26d840939817a203a3ddb0f98f895f4201007f9bb4e3bc9e54a0b0e213f: Status 404 returned error can't find the container with id 79dae26d840939817a203a3ddb0f98f895f4201007f9bb4e3bc9e54a0b0e213f Nov 03 19:49:07.057178 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.057145 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:07.057460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.057414 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:07.061480 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.061389 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:07.192608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.192305 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" event={"ID":"064de3ea-3b4f-483d-8468-62da6e0a19b7","Type":"ContainerStarted","Data":"e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce"} Nov 03 19:49:07.192608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.192374 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" event={"ID":"064de3ea-3b4f-483d-8468-62da6e0a19b7","Type":"ContainerStarted","Data":"79dae26d840939817a203a3ddb0f98f895f4201007f9bb4e3bc9e54a0b0e213f"} Nov 03 19:49:07.236601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.229755 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podStartSLOduration=1.229730652 podStartE2EDuration="1.229730652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:07.22765182 +0000 UTC m=+1054.932491159" watchObservedRunningTime="2026-11-03 19:49:07.229730652 +0000 UTC m=+1054.934569993" Nov 03 19:49:07.724224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:07.724193 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:08.739372 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:08.739339 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:09.209888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:09.209855 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:09.210710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:09.210687 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:49:15.240812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:15.240777 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:49:21.997193 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:49:21.997146 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:49:22.681366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.681336 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2"] Nov 03 19:49:22.692216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.687836 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.692216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.691566 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2"] Nov 03 19:49:22.698479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.698455 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a536f35b-openshift-default-dockercfg-9cctv\"" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813235 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-socket\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813365 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-envoy\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813441 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istiod-ca-cert\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813647 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-certs\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813727 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-token\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813779 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-data\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813810 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-credential-socket\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813887 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cml\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-kube-api-access-57cml\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.821628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.813926 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-podinfo\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.917771 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-data\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.917830 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-credential-socket\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.917919 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cml\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-kube-api-access-57cml\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.917957 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-podinfo\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.918021 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-socket\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.918124 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-envoy\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.918202 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istiod-ca-cert\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.918400 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-certs\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.918489 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-token\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.919289 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-data\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.919649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.919561 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-credential-socket\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.922610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.920868 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-socket\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.922610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.921118 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-certs\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.935670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.928890 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istiod-ca-cert\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.955744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.953521 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cml\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-kube-api-access-57cml\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.956516 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.956450 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-envoy\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.957222 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.957168 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-podinfo\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:22.969333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:22.969151 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-token\") pod \"e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:23.028604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:23.021807 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:23.419520 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:23.419490 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2"] Nov 03 19:49:23.420818 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:49:23.420792 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596a2c8d_09ac_4e12_884b_6b179a74c0b0.slice/crio-0452fcc04fa69877fb444f4e511bf5d739868705f9f59225ca78ab1392d79738 WatchSource:0}: Error finding container 0452fcc04fa69877fb444f4e511bf5d739868705f9f59225ca78ab1392d79738: Status 404 returned error can't find the container with id 0452fcc04fa69877fb444f4e511bf5d739868705f9f59225ca78ab1392d79738 Nov 03 19:49:23.423864 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:23.423832 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:23.423946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:23.423898 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:23.423946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:23.423929 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:24.418602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:24.417735 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" event={"ID":"596a2c8d-09ac-4e12-884b-6b179a74c0b0","Type":"ContainerStarted","Data":"d04dd78fc72ef4351bf080bb4ae3a69e76453616341cc542df87bfa149b09ddf"} Nov 03 19:49:24.418602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:24.417806 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" event={"ID":"596a2c8d-09ac-4e12-884b-6b179a74c0b0","Type":"ContainerStarted","Data":"0452fcc04fa69877fb444f4e511bf5d739868705f9f59225ca78ab1392d79738"} Nov 03 19:49:24.442887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:24.442823 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podStartSLOduration=2.442808951 podStartE2EDuration="2.442808951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:24.439720218 +0000 UTC m=+1072.144559559" watchObservedRunningTime="2026-11-03 19:49:24.442808951 +0000 UTC m=+1072.147648360" Nov 03 19:49:25.024306 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:25.024268 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:25.030239 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:25.030209 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:25.425043 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:25.424958 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:25.426034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:25.426003 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:49:26.510689 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.509272 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn"] Nov 03 19:49:26.515126 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.515089 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn"] Nov 03 19:49:26.515263 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.515243 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.523118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.522736 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fcddeb-openshift-default-dockercfg-cq459\"" Nov 03 19:49:26.633203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633159 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633212 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633335 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q69j\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-kube-api-access-5q69j\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633409 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633437 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633696 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633781 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633734 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633830 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.633957 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.633945 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735175 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735275 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735309 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735345 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q69j\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-kube-api-access-5q69j\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735387 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735413 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735635 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735680 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735736 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.735994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.735974 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.736158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.736133 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.736227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.736171 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.736291 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.736227 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.736516 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.736496 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.738234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.738202 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.738974 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.738952 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.768174 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.768108 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.786376 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.786341 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q69j\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-kube-api-access-5q69j\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:26.851736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:26.851699 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:27.359696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.359666 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn"] Nov 03 19:49:27.366631 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:49:27.361958 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0209b03_fcc6_4a8c_97b8_edefc36eb120.slice/crio-0161cf20dd399a93ed7169c80b11262a37447602036550815ce632c6c55f8c51 WatchSource:0}: Error finding container 0161cf20dd399a93ed7169c80b11262a37447602036550815ce632c6c55f8c51: Status 404 returned error can't find the container with id 0161cf20dd399a93ed7169c80b11262a37447602036550815ce632c6c55f8c51 Nov 03 19:49:27.366631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.365048 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:27.366631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.365118 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:27.366631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.365148 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:27.451547 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.451506 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" event={"ID":"a0209b03-fcc6-4a8c-97b8-edefc36eb120","Type":"ContainerStarted","Data":"aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931"} Nov 03 19:49:27.451690 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.451590 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" event={"ID":"a0209b03-fcc6-4a8c-97b8-edefc36eb120","Type":"ContainerStarted","Data":"0161cf20dd399a93ed7169c80b11262a37447602036550815ce632c6c55f8c51"} Nov 03 19:49:27.474958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.473908 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podStartSLOduration=1.473886995 podStartE2EDuration="1.473886995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:27.471644238 +0000 UTC m=+1075.176483577" watchObservedRunningTime="2026-11-03 19:49:27.473886995 +0000 UTC m=+1075.178726335" Nov 03 19:49:27.858719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:27.858682 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:28.853840 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:28.853804 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:49:28.854005 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:28.853859 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:49:29.852094 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:29.852061 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:49:29.852504 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:29.852114 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:49:30.852467 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:30.852419 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:49:30.852970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:30.852490 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:49:31.852451 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:31.852412 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:49:31.852718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:31.852475 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:49:32.852619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:32.852570 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:49:32.852824 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:32.852657 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:49:33.852768 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:33.852721 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:49:33.852962 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:33.852794 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:49:34.860121 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:34.860081 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:34.861122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:34.861096 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:34.866873 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:34.866845 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 19:49:36.222844 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.222807 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt"] Nov 03 19:49:36.245929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.245900 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt"] Nov 03 19:49:36.246147 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.246015 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.256256 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.256230 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a536f35b-serving-cert\"" Nov 03 19:49:36.418444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.417280 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-metrics-certs\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.418444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.417355 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtbx\" (UniqueName: \"kubernetes.io/projected/0b67ac80-3318-49e9-a660-0a4c37f14483-kube-api-access-7rtbx\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.418444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.417502 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.418444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.417542 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b67ac80-3318-49e9-a660-0a4c37f14483-openshift-service-ca\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.519335 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.519299 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.519546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.519345 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b67ac80-3318-49e9-a660-0a4c37f14483-openshift-service-ca\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.519546 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:49:36.519454 4314 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-serving-cert: secret "maas-api-e2e-mt-a536f35b-serving-cert" not found Nov 03 19:49:36.519546 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:49:36.519525 4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls podName:0b67ac80-3318-49e9-a660-0a4c37f14483 nodeName:}" failed. No retries permitted until 2026-11-03 19:49:37.019508147 +0000 UTC m=+1084.724347480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls") pod "maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" (UID: "0b67ac80-3318-49e9-a660-0a4c37f14483") : secret "maas-api-e2e-mt-a536f35b-serving-cert" not found Nov 03 19:49:36.519776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.519670 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-metrics-certs\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.519776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.519708 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtbx\" (UniqueName: \"kubernetes.io/projected/0b67ac80-3318-49e9-a660-0a4c37f14483-kube-api-access-7rtbx\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.520164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.520141 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b67ac80-3318-49e9-a660-0a4c37f14483-openshift-service-ca\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.522298 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.522275 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-metrics-certs\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:36.528476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:36.528452 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtbx\" (UniqueName: \"kubernetes.io/projected/0b67ac80-3318-49e9-a660-0a4c37f14483-kube-api-access-7rtbx\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:37.036502 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:37.036468 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:37.038945 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:37.038922 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls\") pod \"maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:37.171412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:37.171381 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:37.506189 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:37.506146 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt"] Nov 03 19:49:37.515608 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:49:37.515568 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b67ac80_3318_49e9_a660_0a4c37f14483.slice/crio-dd216308b9107148c1a5c43001a6e741da78ee3c8d34a8d37a25871db1127261 WatchSource:0}: Error finding container dd216308b9107148c1a5c43001a6e741da78ee3c8d34a8d37a25871db1127261: Status 404 returned error can't find the container with id dd216308b9107148c1a5c43001a6e741da78ee3c8d34a8d37a25871db1127261 Nov 03 19:49:37.558330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:37.558293 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" event={"ID":"0b67ac80-3318-49e9-a660-0a4c37f14483","Type":"ContainerStarted","Data":"dd216308b9107148c1a5c43001a6e741da78ee3c8d34a8d37a25871db1127261"} Nov 03 19:49:38.570815 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:38.570776 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" event={"ID":"0b67ac80-3318-49e9-a660-0a4c37f14483","Type":"ContainerStarted","Data":"76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2"} Nov 03 19:49:38.571378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:38.571352 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:44.587228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:44.587200 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:49:44.606281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:44.606236 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" podStartSLOduration=8.111374627 podStartE2EDuration="8.606223361s" totalImagesPullingTime="494.848734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:38.588749989 +0000 UTC m=+1086.293589330" watchObservedRunningTime="2026-11-03 19:49:44.606223361 +0000 UTC m=+1092.311062701" Nov 03 19:49:52.181607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:52.179818 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:52.231380 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:49:52.231310 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:49:52.339699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:52.339653 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:49:52.365714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:52.365672 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:52.371670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:52.371637 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:49:52.555615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:52.545683 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.184717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.184664 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.185890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.185863 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:49:53.198684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.198649 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.362604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.359642 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.362604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.360421 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:49:53.464673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.423264 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:49:53.464673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.425209 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:49:53.464673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.426261 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.464673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.426632 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.464673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.430035 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.464673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.430881 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:49:53.602710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.549263 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:53.602710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:53.551059 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:49:54.273618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.269851 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:54.360644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.346078 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 03 19:49:54.361720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.361670 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:54.399615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.396534 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:49:54.407608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.401151 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:49:54.435233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.435198 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:54.592107 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.591998 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:54.615854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:54.615795 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:49:55.029829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.029782 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-76cxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:49:55.030280 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.030237 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:55.046823 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.046763 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:49:55.313115 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.313019 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:49:55.409511 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.405735 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 19:49:55.487997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.477985 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:55.504797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.494747 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Nov 03 19:49:55.518340 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.518293 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:55.558251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.515647 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:49:55.653695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.639303 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:49:55.653695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.639481 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:55.717932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.717889 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:55.993822 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:55.993792 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:49:56.055295 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.055267 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:49:56.062036 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.062011 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:49:56.351184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.337027 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:49:56.351184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.341160 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:56.351184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.342687 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:49:56.351184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.342763 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:49:56.351184 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.344482 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:49:56.454958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.454931 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:49:56.621670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.616663 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp"] Nov 03 19:49:56.638757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.638725 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.652264 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.651933 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp"] Nov 03 19:49:56.655817 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.655774 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.655817 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.655812 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.656015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.655837 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.656015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.655858 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5452c\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-kube-api-access-5452c\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.656015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.655907 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.656015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.655932 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.656015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.655991 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.656015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.656017 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.656317 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.656189 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.821288 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.821248 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.821613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.821573 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.821772 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.821749 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.825608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.821881 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.825787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.825765 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5452c\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-kube-api-access-5452c\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.825970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.825954 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.826602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.826122 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.829921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.829896 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.830190 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.830168 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.830434 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.830413 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.830606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.830563 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.830962 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.830944 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.831147 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.831130 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.831269 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.831255 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.831953 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.831929 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.833561 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.833540 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.840684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.840658 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:56.844431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:56.844250 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5452c\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-kube-api-access-5452c\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:57.050527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:57.050489 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:57.540480 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:57.540423 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:57.540947 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:57.540535 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:57.540947 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:57.540617 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:49:57.549936 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:57.549695 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp"] Nov 03 19:49:58.027832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:58.027785 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" event={"ID":"9dfd69b0-5b9c-4e8e-81ae-700825b0b858","Type":"ContainerStarted","Data":"40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75"} Nov 03 19:49:58.027999 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:58.027868 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" event={"ID":"9dfd69b0-5b9c-4e8e-81ae-700825b0b858","Type":"ContainerStarted","Data":"9a353f47e03e1c65ed77e6d371445e5ea3fb80945559431357a42c80cd392e2c"} Nov 03 19:49:58.051789 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:58.049397 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podStartSLOduration=2.049379966 podStartE2EDuration="2.049379966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:58.047547567 +0000 UTC m=+1105.752386908" watchObservedRunningTime="2026-11-03 19:49:58.049379966 +0000 UTC m=+1105.754219307" Nov 03 19:49:58.054783 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:58.053471 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:49:58.055925 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:58.055602 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:49:58.055925 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:58.055654 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:49:59.051791 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.051750 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:49:59.052181 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.051817 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:49:59.138874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.138724 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:49:59.138874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.138785 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:49:59.620503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.620468 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9"] Nov 03 19:49:59.644338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.644154 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.649939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.649532 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9"] Nov 03 19:49:59.669396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.668878 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3c7998-openshift-default-dockercfg-b2qh7\"" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.725625 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-socket\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.725728 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-envoy\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.725805 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-certs\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.725839 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-data\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.726033 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbbz\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-kube-api-access-dtbbz\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.726148 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-credential-socket\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.726232 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b551235-3802-4549-9657-8cb3cc22e12b-istiod-ca-cert\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.726260 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b551235-3802-4549-9657-8cb3cc22e12b-istio-podinfo\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.727173 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.726309 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-istio-token\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.828235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.828167 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-credential-socket\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.828424 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.828265 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b551235-3802-4549-9657-8cb3cc22e12b-istiod-ca-cert\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.828424 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.828309 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b551235-3802-4549-9657-8cb3cc22e12b-istio-podinfo\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.828424 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.828361 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-istio-token\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.842668 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.828547 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-socket\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.867916 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.862157 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b551235-3802-4549-9657-8cb3cc22e12b-istiod-ca-cert\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.902829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.901469 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-envoy\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.902829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.901560 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-certs\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.902829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.901613 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-data\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.902829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.901854 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbbz\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-kube-api-access-dtbbz\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.920602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.904192 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-certs\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.920602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.906931 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-data\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.920602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.916750 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-socket\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.928602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.923099 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-credential-socket\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.980230 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.980156 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b551235-3802-4549-9657-8cb3cc22e12b-istio-podinfo\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.982729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.982676 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbbz\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-kube-api-access-dtbbz\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:49:59.986319 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:49:59.986272 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-istio-token\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:00.031988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.031917 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-envoy\") pod \"e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:00.058604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.056932 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:50:00.058604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.056990 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:50:00.310632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.310360 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:00.834873 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.834836 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9"] Nov 03 19:50:00.845157 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:50:00.845129 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b551235_3802_4549_9657_8cb3cc22e12b.slice/crio-11a195ac9fa0ed9af8180123d67155d29c10a6e5882dca76640af9d0a2af9203 WatchSource:0}: Error finding container 11a195ac9fa0ed9af8180123d67155d29c10a6e5882dca76640af9d0a2af9203: Status 404 returned error can't find the container with id 11a195ac9fa0ed9af8180123d67155d29c10a6e5882dca76640af9d0a2af9203 Nov 03 19:50:00.848648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.848609 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:00.848755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.848704 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:00.848755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:00.848748 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:01.051287 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:01.051252 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:50:01.051455 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:01.051307 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:50:01.111744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:01.111639 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" event={"ID":"5b551235-3802-4549-9657-8cb3cc22e12b","Type":"ContainerStarted","Data":"f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f"} Nov 03 19:50:01.111744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:01.111714 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" event={"ID":"5b551235-3802-4549-9657-8cb3cc22e12b","Type":"ContainerStarted","Data":"11a195ac9fa0ed9af8180123d67155d29c10a6e5882dca76640af9d0a2af9203"} Nov 03 19:50:01.133527 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:01.133471 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" podStartSLOduration=2.133454714 podStartE2EDuration="2.133454714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:01.132478703 +0000 UTC m=+1108.837318058" watchObservedRunningTime="2026-11-03 19:50:01.133454714 +0000 UTC m=+1108.838294055" Nov 03 19:50:01.315794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:01.315687 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:02.051877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:02.051839 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:50:02.052072 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:02.051905 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:50:02.320214 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:02.320131 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:03.051776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:03.051732 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:50:03.052035 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:03.051819 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:50:03.127837 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:03.127804 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:03.129190 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:03.129167 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:04.052705 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:04.052105 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 19:50:04.052705 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:04.052239 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 19:50:05.193603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:05.192865 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:50:05.200687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:05.200653 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:50:05.241983 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:05.241952 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:50:06.245535 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:06.245503 4314 generic.go:356] "Generic (PLEG): container finished" podID="e508fa96-3607-45d5-ad64-e806599da0a7" containerID="536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484" exitCode=0 Nov 03 19:50:06.246067 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:06.245591 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" event={"ID":"e508fa96-3607-45d5-ad64-e806599da0a7","Type":"ContainerDied","Data":"536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484"} Nov 03 19:50:06.246183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:06.246165 4314 scope.go:122] "RemoveContainer" containerID="536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484" Nov 03 19:50:06.247978 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:06.247952 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:06.248054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:06.248017 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:06.248054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:06.248049 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:07.344926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:07.344839 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" event={"ID":"e508fa96-3607-45d5-ad64-e806599da0a7","Type":"ContainerStarted","Data":"bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed"} Nov 03 19:50:07.877604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:07.876939 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:50:07.887657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:07.882815 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:50:07.887657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:07.882914 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 19:50:08.352950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.352919 4314 generic.go:356] "Generic (PLEG): container finished" podID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerID="e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce" exitCode=0 Nov 03 19:50:08.353472 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.352988 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" event={"ID":"064de3ea-3b4f-483d-8468-62da6e0a19b7","Type":"ContainerDied","Data":"e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce"} Nov 03 19:50:08.353551 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.353530 4314 scope.go:122] "RemoveContainer" containerID="e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce" Nov 03 19:50:08.355441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.355406 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:08.355560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.355481 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:08.355560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.355517 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:08.873602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.872948 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:50:08.873602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:08.873010 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 19:50:09.214704 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:09.212837 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:50:09.422415 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:09.422369 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" event={"ID":"064de3ea-3b4f-483d-8468-62da6e0a19b7","Type":"ContainerStarted","Data":"dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e"} Nov 03 19:50:09.732605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:09.727785 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:50:09.740605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:09.737173 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:09.740605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:09.737246 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:50:09.875608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:09.872951 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:50:09.875608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:09.873028 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 19:50:10.742602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:10.741694 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:10.742602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:10.741763 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:50:10.876602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:10.875941 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:50:10.876602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:10.876041 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 19:50:11.725350 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:11.725319 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:11.725533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:11.725377 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:50:11.876606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:11.875977 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 19:50:11.876606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:11.876044 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 19:50:12.725320 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:12.725272 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:12.725541 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:12.725353 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:50:12.896003 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:12.895969 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:50:12.896988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:12.896955 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:50:12.897963 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:12.897936 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 19:50:13.725445 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:13.725403 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:13.725692 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:13.725465 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:50:14.730221 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:14.730186 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 19:50:14.730830 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:14.730797 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 19:50:15.729965 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:15.729937 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:50:15.730474 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:15.730459 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:50:15.731282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:15.731260 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 19:50:22.031515 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:50:22.031471 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:50:28.577984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:28.577953 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:50:33.362718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:33.362672 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:33.458355 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:33.458299 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:33.619480 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:33.618083 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:50:33.619480 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:33.618147 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:33.646278 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:33.642664 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:50:33.704603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:33.686723 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-57ktj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:50:33.704603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:33.686815 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:34.373605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:34.371465 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:50:34.460615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:34.457041 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:50:37.864419 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:37.860681 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb"] Nov 03 19:50:37.895664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:37.895625 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb"] Nov 03 19:50:37.895907 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:37.895757 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:37.928466 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:37.928435 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b73c810-openshift-default-dockercfg-mm4nn\"" Nov 03 19:50:38.042492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.042454 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-podinfo\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.042696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.042568 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-envoy\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.042696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.042638 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-token\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.042797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.042758 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtzhn\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-kube-api-access-mtzhn\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.042839 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.042818 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istiod-ca-cert\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.043087 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.043071 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-credential-socket\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.043159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.043122 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-socket\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.043235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.043212 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-certs\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.043339 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.043322 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-data\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.144629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.144549 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-credential-socket\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.144821 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.144639 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-socket\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.144821 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.144700 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-certs\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.144821 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.144810 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-data\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.145020 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.144996 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-podinfo\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.145110 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.145091 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-envoy\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.145241 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.145126 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-token\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.145341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.145241 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtzhn\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-kube-api-access-mtzhn\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.145341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.145268 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istiod-ca-cert\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.145481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.145454 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-socket\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.146091 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.145770 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-credential-socket\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.146091 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.146023 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-certs\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.146091 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.146068 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istiod-ca-cert\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.146467 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.146445 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-data\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.148676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.148641 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-envoy\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.148958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.148905 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-podinfo\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.155839 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.155814 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-token\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.179599 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.156362 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtzhn\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-kube-api-access-mtzhn\") pod \"e2e-mt-2b73c810-openshift-default-68d568696f-twzgb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.250371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.250330 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:38.592796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.590713 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb"] Nov 03 19:50:38.615654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.613109 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:38.615654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.613216 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:38.615654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:38.613267 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:39.484298 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:39.484257 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" event={"ID":"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb","Type":"ContainerStarted","Data":"2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71"} Nov 03 19:50:39.484749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:39.484331 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" event={"ID":"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb","Type":"ContainerStarted","Data":"bf1308fdef435c77a8213bbcc7c2bf3a5eb3788d41c181e820af09e671d3110f"} Nov 03 19:50:39.509155 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:39.509101 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podStartSLOduration=2.509086702 podStartE2EDuration="2.509086702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:39.50500023 +0000 UTC m=+1147.209839571" watchObservedRunningTime="2026-11-03 19:50:39.509086702 +0000 UTC m=+1147.213926041" Nov 03 19:50:40.251127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:40.251087 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:40.258792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:40.258748 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:40.500812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:40.500639 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:40.502371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:40.502317 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:50:44.738029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:44.737991 4314 generic.go:356] "Generic (PLEG): container finished" podID="5b551235-3802-4549-9657-8cb3cc22e12b" containerID="f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f" exitCode=0 Nov 03 19:50:44.738568 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:44.738030 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" event={"ID":"5b551235-3802-4549-9657-8cb3cc22e12b","Type":"ContainerDied","Data":"f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f"} Nov 03 19:50:44.738694 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:44.738679 4314 scope.go:122] "RemoveContainer" containerID="f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f" Nov 03 19:50:44.741013 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:44.740977 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:44.741112 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:44.741073 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:44.741154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:44.741119 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:50:45.799039 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:45.798991 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" event={"ID":"5b551235-3802-4549-9657-8cb3cc22e12b","Type":"ContainerStarted","Data":"b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350"} Nov 03 19:50:46.311917 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:46.311876 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:46.312997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:46.312964 4314 patch_prober.go:29] interesting pod/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:50:46.313241 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:46.313208 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:50:47.311364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:47.311321 4314 patch_prober.go:29] interesting pod/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:50:47.311781 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:47.311391 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:50:48.128065 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:48.128029 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:48.311959 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:48.311916 4314 patch_prober.go:29] interesting pod/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:50:48.312474 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:48.311994 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:50:49.311261 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:49.311223 4314 patch_prober.go:29] interesting pod/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:50:49.311426 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:49.311275 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:50:50.311974 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:50.311940 4314 patch_prober.go:29] interesting pod/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 19:50:50.312466 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:50.311994 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 19:50:51.317254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:51.317222 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:51.318695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:51.318667 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:50:52.025239 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:50:52.025168 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:50:53.934504 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:53.934463 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m"] Nov 03 19:50:53.938962 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:53.938915 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:53.942961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:53.942919 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-fa9526-serving-cert\"" Nov 03 19:50:53.978074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:53.976725 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m"] Nov 03 19:50:54.022117 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.022075 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.022692 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.022655 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f624939-9997-46de-81ea-11d18972084d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.023672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.023648 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86tz\" (UniqueName: \"kubernetes.io/projected/6f624939-9997-46de-81ea-11d18972084d-kube-api-access-p86tz\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.023989 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.023970 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.153516 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.144957 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.171602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.157840 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.173570 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.173536 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f624939-9997-46de-81ea-11d18972084d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.175007 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.174980 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86tz\" (UniqueName: \"kubernetes.io/projected/6f624939-9997-46de-81ea-11d18972084d-kube-api-access-p86tz\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.176245 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:50:54.145772 4314 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-serving-cert: secret "maas-api-e2e-shared-a-w2-fa9526-serving-cert" not found Nov 03 19:50:54.176458 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:50:54.176446 4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls podName:6f624939-9997-46de-81ea-11d18972084d nodeName:}" failed. No retries permitted until 2026-11-03 19:50:54.676422578 +0000 UTC m=+1162.381261911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" (UID: "6f624939-9997-46de-81ea-11d18972084d") : secret "maas-api-e2e-shared-a-w2-fa9526-serving-cert" not found Nov 03 19:50:54.176794 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.172311 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.176943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.174784 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f624939-9997-46de-81ea-11d18972084d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.212410 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.212303 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86tz\" (UniqueName: \"kubernetes.io/projected/6f624939-9997-46de-81ea-11d18972084d-kube-api-access-p86tz\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.705608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.705107 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.727474 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.727439 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:54.917501 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:54.917204 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:55.452673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:55.452319 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m"] Nov 03 19:50:56.027968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:56.027935 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" event={"ID":"6f624939-9997-46de-81ea-11d18972084d","Type":"ContainerStarted","Data":"778be5d75b5998cbd1a4f28fc87dade7fb41a59249f09eca8bc4604c78f803a1"} Nov 03 19:50:57.044407 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:57.044374 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" event={"ID":"6f624939-9997-46de-81ea-11d18972084d","Type":"ContainerStarted","Data":"b7615c6a31e805dc029d8b461ba29a818f7c209b19d8eccdf2682b5344d2409f"} Nov 03 19:50:57.045020 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:57.045004 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:50:57.077139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:50:57.077043 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" podStartSLOduration=3.538198168 podStartE2EDuration="4.077023353s" totalImagesPullingTime="538.825185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:57.062286893 +0000 UTC m=+1164.767126234" watchObservedRunningTime="2026-11-03 19:50:57.077023353 +0000 UTC m=+1164.781862694" Nov 03 19:51:03.092654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:03.092621 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 19:51:13.685007 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:13.684973 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:51:22.004566 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:51:22.004526 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:51:26.246133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.245458 4314 patch_prober.go:29] interesting pod/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": read tcp 10.134.0.2:35014->10.134.0.101:15021: read: connection reset by peer" start-of-body= Nov 03 19:51:26.246133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.245526 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": read tcp 10.134.0.2:35014->10.134.0.101:15021: read: connection reset by peer" Nov 03 19:51:26.567071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.566987 4314 generic.go:356] "Generic (PLEG): container finished" podID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerID="2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71" exitCode=0 Nov 03 19:51:26.567071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.567042 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" event={"ID":"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb","Type":"ContainerDied","Data":"2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71"} Nov 03 19:51:26.567513 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.567492 4314 scope.go:122] "RemoveContainer" containerID="2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71" Nov 03 19:51:26.569539 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.569494 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:51:26.569680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.569614 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:51:26.569680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:26.569664 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:51:27.601542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:27.601496 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" event={"ID":"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb","Type":"ContainerStarted","Data":"557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526"} Nov 03 19:51:28.250693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:28.250657 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:51:28.250946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:28.250926 4314 patch_prober.go:29] interesting pod/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:51:28.251021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:28.250971 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 19:51:29.251567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:29.251509 4314 patch_prober.go:29] interesting pod/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:51:29.251567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:29.251576 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 19:51:30.251593 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:30.251543 4314 patch_prober.go:29] interesting pod/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:51:30.251764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:30.251633 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 19:51:31.251488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:31.251448 4314 patch_prober.go:29] interesting pod/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:51:31.251667 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:31.251504 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 19:51:32.250919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:32.250881 4314 patch_prober.go:29] interesting pod/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:51:32.251322 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:32.250951 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 19:51:32.930280 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:32.930234 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:51:32.932503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:32.932476 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:51:33.251076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:33.251041 4314 patch_prober.go:29] interesting pod/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 19:51:33.251491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:33.251109 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 19:51:34.255864 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:34.255832 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:51:34.256302 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:34.256280 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:51:34.257414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:34.257390 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:51:48.343779 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.343683 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46"] Nov 03 19:51:48.352963 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.352934 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.366398 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.366354 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9f84de-openshift-default-dockercfg-qhn66\"" Nov 03 19:51:48.373005 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.372967 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46"] Nov 03 19:51:48.452827 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.452796 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-socket\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.452977 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.452859 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-istio-token\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.453048 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.453025 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38088147-4b05-4987-82b0-85f536cd2697-istio-podinfo\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.453090 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.453065 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-envoy\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.453215 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.453195 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-kube-api-access-795z9\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.453250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.453231 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-data\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.453284 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.453251 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-certs\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.453339 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.453326 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-credential-socket\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.453384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.453373 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38088147-4b05-4987-82b0-85f536cd2697-istiod-ca-cert\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568132 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-istio-token\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568385 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38088147-4b05-4987-82b0-85f536cd2697-istio-podinfo\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568429 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-envoy\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568575 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-kube-api-access-795z9\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568639 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-data\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568670 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-certs\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568701 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-credential-socket\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568782 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38088147-4b05-4987-82b0-85f536cd2697-istiod-ca-cert\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.569612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.568966 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-socket\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.573603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.572526 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-data\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.573603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.573023 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-credential-socket\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.573603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.573273 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-socket\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.577608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.573885 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38088147-4b05-4987-82b0-85f536cd2697-istiod-ca-cert\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.577608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.574262 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-certs\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.577608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.577149 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38088147-4b05-4987-82b0-85f536cd2697-istio-podinfo\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.592600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.579642 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-envoy\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.610695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.607005 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-kube-api-access-795z9\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.624338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.624276 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-istio-token\") pod \"e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:48.701782 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:48.701748 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:49.223972 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:49.223938 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46"] Nov 03 19:51:49.224424 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:51:49.224400 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38088147_4b05_4987_82b0_85f536cd2697.slice/crio-d5c752167b24d93cb3a89b324fe319ed7e053d5c506b015c2303cb173aa16388 WatchSource:0}: Error finding container d5c752167b24d93cb3a89b324fe319ed7e053d5c506b015c2303cb173aa16388: Status 404 returned error can't find the container with id d5c752167b24d93cb3a89b324fe319ed7e053d5c506b015c2303cb173aa16388 Nov 03 19:51:49.227489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:49.227453 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:51:49.227572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:49.227520 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:51:49.227572 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:49.227549 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:51:49.934672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:49.934635 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" event={"ID":"38088147-4b05-4987-82b0-85f536cd2697","Type":"ContainerStarted","Data":"e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697"} Nov 03 19:51:49.935078 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:49.934690 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" event={"ID":"38088147-4b05-4987-82b0-85f536cd2697","Type":"ContainerStarted","Data":"d5c752167b24d93cb3a89b324fe319ed7e053d5c506b015c2303cb173aa16388"} Nov 03 19:51:49.957891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:49.957836 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podStartSLOduration=1.9578249699999999 podStartE2EDuration="1.95782497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:49.955673815 +0000 UTC m=+1217.660513156" watchObservedRunningTime="2026-11-03 19:51:49.95782497 +0000 UTC m=+1217.662664310" Nov 03 19:51:50.706604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:50.702547 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:50.715695 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:50.715659 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:50.942519 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:50.942488 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:50.943408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:50.943390 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:51:51.994263 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:51:51.994224 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:51:54.257226 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:54.257171 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:51:59.858240 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:59.858202 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9"] Nov 03 19:51:59.860590 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:51:59.860542 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" containerID="cri-o://b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350" gracePeriod=30 Nov 03 19:52:05.192441 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.192409 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:52:05.316181 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316097 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-envoy\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316181 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316152 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-socket\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316296 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-credential-socket\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316353 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-certs\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316452 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-data\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316492 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-socket" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:05.316656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316526 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b551235-3802-4549-9657-8cb3cc22e12b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b551235-3802-4549-9657-8cb3cc22e12b-istio-podinfo\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316631 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-credential-socket" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:05.316851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316672 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-istio-token\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-istio-token\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316691 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b551235-3802-4549-9657-8cb3cc22e12b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b551235-3802-4549-9657-8cb3cc22e12b-istiod-ca-cert\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316719 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-certs" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:05.316851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316734 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-kube-api-access-dtbbz\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-kube-api-access-dtbbz\") pod \"5b551235-3802-4549-9657-8cb3cc22e12b\" (UID: \"5b551235-3802-4549-9657-8cb3cc22e12b\") " Nov 03 19:52:05.316851 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.316731 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-data" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:05.317143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.317039 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b551235-3802-4549-9657-8cb3cc22e12b-istiod-ca-cert" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:05.317646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.317618 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b551235-3802-4549-9657-8cb3cc22e12b-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.317713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.317645 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.317713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.317658 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.317713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.317667 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.317713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.317675 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.319654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.319533 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-kube-api-access-dtbbz" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "kube-api-access-dtbbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:05.319759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.319652 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b551235-3802-4549-9657-8cb3cc22e12b-istio-podinfo" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:52:05.319759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.319672 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-envoy" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:05.319759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.319708 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-istio-token" pod "5b551235-3802-4549-9657-8cb3cc22e12b" (UID: "5b551235-3802-4549-9657-8cb3cc22e12b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:05.419364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.419331 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b551235-3802-4549-9657-8cb3cc22e12b-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.419364 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.419362 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b551235-3802-4549-9657-8cb3cc22e12b-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.419558 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.419373 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.419558 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.419384 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtbbz\" (UniqueName: \"kubernetes.io/projected/5b551235-3802-4549-9657-8cb3cc22e12b-kube-api-access-dtbbz\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:05.536152 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.536119 4314 generic.go:356] "Generic (PLEG): container finished" podID="5b551235-3802-4549-9657-8cb3cc22e12b" containerID="b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350" exitCode=0 Nov 03 19:52:05.536312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.536199 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" event={"ID":"5b551235-3802-4549-9657-8cb3cc22e12b","Type":"ContainerDied","Data":"b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350"} Nov 03 19:52:05.536312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.536297 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" event={"ID":"5b551235-3802-4549-9657-8cb3cc22e12b","Type":"ContainerDied","Data":"11a195ac9fa0ed9af8180123d67155d29c10a6e5882dca76640af9d0a2af9203"} Nov 03 19:52:05.536389 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.536332 4314 scope.go:122] "RemoveContainer" containerID="b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350" Nov 03 19:52:05.538042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.538018 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9" Nov 03 19:52:05.555183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.555159 4314 scope.go:122] "RemoveContainer" containerID="f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f" Nov 03 19:52:05.584248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.583867 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9"] Nov 03 19:52:05.589210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.589188 4314 scope.go:122] "RemoveContainer" containerID="b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350" Nov 03 19:52:05.589827 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:52:05.589799 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350\": container with ID starting with b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350 not found: ID does not exist" containerID="b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350" Nov 03 19:52:05.589934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.589832 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350"} err="failed to get container status \"b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350\": rpc error: code = NotFound desc = could not find container \"b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350\": container with ID starting with b66c7c8ca6a8e1708ff2039b8d2728da1cf33a98975c89c4be069470d89ee350 not found: ID does not exist" Nov 03 19:52:05.589934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.589853 4314 scope.go:122] "RemoveContainer" containerID="f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f" Nov 03 19:52:05.590115 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:52:05.590081 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f\": container with ID starting with f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f not found: ID does not exist" containerID="f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f" Nov 03 19:52:05.590115 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.590105 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f"} err="failed to get container status \"f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f\": rpc error: code = NotFound desc = could not find container \"f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f\": container with ID starting with f38145196bd5e1357e007e6ac6030e5b34cf05df95eb43941050157bf88fd55f not found: ID does not exist" Nov 03 19:52:05.591902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:05.591876 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3c7998-openshift-default-cf99794c5-44cv9"] Nov 03 19:52:06.430457 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:06.430423 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" path="/var/lib/kubelet/pods/5b551235-3802-4549-9657-8cb3cc22e12b/volumes" Nov 03 19:52:17.050642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:17.050612 4314 generic.go:356] "Generic (PLEG): container finished" podID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerID="d04dd78fc72ef4351bf080bb4ae3a69e76453616341cc542df87bfa149b09ddf" exitCode=0 Nov 03 19:52:17.051025 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:17.050685 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" event={"ID":"596a2c8d-09ac-4e12-884b-6b179a74c0b0","Type":"ContainerDied","Data":"d04dd78fc72ef4351bf080bb4ae3a69e76453616341cc542df87bfa149b09ddf"} Nov 03 19:52:17.051126 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:17.051112 4314 scope.go:122] "RemoveContainer" containerID="d04dd78fc72ef4351bf080bb4ae3a69e76453616341cc542df87bfa149b09ddf" Nov 03 19:52:17.053149 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:17.053117 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:17.053262 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:17.053195 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:17.053262 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:17.053229 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:18.061462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:18.061426 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" event={"ID":"596a2c8d-09ac-4e12-884b-6b179a74c0b0","Type":"ContainerStarted","Data":"b51ce5b1de05d0c87a13caef50576f5c15e29409badddeedd2bec40c0976b66a"} Nov 03 19:52:19.024215 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:19.024182 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:52:19.024427 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:19.024287 4314 patch_prober.go:29] interesting pod/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:52:19.024427 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:19.024323 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:52:20.024571 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:20.024528 4314 patch_prober.go:29] interesting pod/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:52:20.025121 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:20.024633 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:52:21.025999 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:21.025960 4314 patch_prober.go:29] interesting pod/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:52:21.026615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:21.026554 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:52:22.043051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:22.043010 4314 patch_prober.go:29] interesting pod/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:52:22.043815 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:22.043777 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:52:22.281625 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:52:22.279329 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:52:22.359664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:22.358886 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:22.400609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:22.396873 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:22.427605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:22.417711 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:22.540608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:22.540422 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:52:22.702763 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:22.702660 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.026605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.026034 4314 patch_prober.go:29] interesting pod/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:52:23.026605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.026125 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:52:23.120781 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.120718 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.361799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.361693 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.362656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.362374 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:52:23.413916 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.413866 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded" Nov 03 19:52:23.433317 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.432621 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:52:23.433317 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.433194 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:52:23.452064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.452004 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:52:23.529100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:23.529068 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:52:24.026064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:24.026020 4314 patch_prober.go:29] interesting pod/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 19:52:24.026358 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:24.026326 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 19:52:25.088082 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:25.088049 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:52:25.088638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:25.088617 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:52:25.090617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:25.090594 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:52:33.858905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:33.858317 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm"] Nov 03 19:52:33.862677 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:33.862642 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" containerID="cri-o://19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b" gracePeriod=30 Nov 03 19:52:36.510361 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:36.510321 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt"] Nov 03 19:52:36.512007 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:36.511747 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" podUID="0b67ac80-3318-49e9-a660-0a4c37f14483" containerName="maas-api" containerID="cri-o://76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2" gracePeriod=30 Nov 03 19:52:37.221332 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.221301 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:52:37.293604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.284626 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-metrics-certs\") pod \"0b67ac80-3318-49e9-a660-0a4c37f14483\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " Nov 03 19:52:37.294493 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.294463 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls\") pod \"0b67ac80-3318-49e9-a660-0a4c37f14483\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " Nov 03 19:52:37.294939 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.294919 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b67ac80-3318-49e9-a660-0a4c37f14483-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b67ac80-3318-49e9-a660-0a4c37f14483-openshift-service-ca\") pod \"0b67ac80-3318-49e9-a660-0a4c37f14483\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " Nov 03 19:52:37.295216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.295189 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b67ac80-3318-49e9-a660-0a4c37f14483-kube-api-access-7rtbx\" (UniqueName: \"kubernetes.io/projected/0b67ac80-3318-49e9-a660-0a4c37f14483-kube-api-access-7rtbx\") pod \"0b67ac80-3318-49e9-a660-0a4c37f14483\" (UID: \"0b67ac80-3318-49e9-a660-0a4c37f14483\") " Nov 03 19:52:37.296638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.296609 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b67ac80-3318-49e9-a660-0a4c37f14483-openshift-service-ca" pod "0b67ac80-3318-49e9-a660-0a4c37f14483" (UID: "0b67ac80-3318-49e9-a660-0a4c37f14483"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:37.297666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.297643 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b67ac80-3318-49e9-a660-0a4c37f14483-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:37.327903 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.327558 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b67ac80-3318-49e9-a660-0a4c37f14483-kube-api-access-7rtbx" pod "0b67ac80-3318-49e9-a660-0a4c37f14483" (UID: "0b67ac80-3318-49e9-a660-0a4c37f14483"). InnerVolumeSpecName "kube-api-access-7rtbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:37.327903 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.327775 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-metrics-certs" pod "0b67ac80-3318-49e9-a660-0a4c37f14483" (UID: "0b67ac80-3318-49e9-a660-0a4c37f14483"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:37.328040 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.327936 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls" pod "0b67ac80-3318-49e9-a660-0a4c37f14483" (UID: "0b67ac80-3318-49e9-a660-0a4c37f14483"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:52:37.409609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.409492 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:37.409609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.409529 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b67ac80-3318-49e9-a660-0a4c37f14483-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:37.409609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.409545 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtbx\" (UniqueName: \"kubernetes.io/projected/0b67ac80-3318-49e9-a660-0a4c37f14483-kube-api-access-7rtbx\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:37.462564 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.462476 4314 generic.go:356] "Generic (PLEG): container finished" podID="0b67ac80-3318-49e9-a660-0a4c37f14483" containerID="76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2" exitCode=0 Nov 03 19:52:37.462564 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.462552 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" event={"ID":"0b67ac80-3318-49e9-a660-0a4c37f14483","Type":"ContainerDied","Data":"76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2"} Nov 03 19:52:37.462788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.462623 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" event={"ID":"0b67ac80-3318-49e9-a660-0a4c37f14483","Type":"ContainerDied","Data":"dd216308b9107148c1a5c43001a6e741da78ee3c8d34a8d37a25871db1127261"} Nov 03 19:52:37.462788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.462661 4314 scope.go:122] "RemoveContainer" containerID="76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2" Nov 03 19:52:37.462867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.462841 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt" Nov 03 19:52:37.494017 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.491347 4314 scope.go:122] "RemoveContainer" containerID="76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2" Nov 03 19:52:37.494017 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:52:37.491837 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2\": container with ID starting with 76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2 not found: ID does not exist" containerID="76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2" Nov 03 19:52:37.494017 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.491872 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2"} err="failed to get container status \"76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2\": rpc error: code = NotFound desc = could not find container \"76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2\": container with ID starting with 76c9d1a890c244007f841c0fafc25ad8e7f2c288408654480ad3d7039053b7a2 not found: ID does not exist" Nov 03 19:52:37.522604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.509835 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt"] Nov 03 19:52:37.522604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:37.518504 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a536f35b-5c4c8fb785-n7zxt"] Nov 03 19:52:38.442612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:38.440951 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b67ac80-3318-49e9-a660-0a4c37f14483" path="/var/lib/kubelet/pods/0b67ac80-3318-49e9-a660-0a4c37f14483/volumes" Nov 03 19:52:38.749674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:38.738895 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:52:38.749674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:38.738962 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:52:39.226156 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.226128 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:52:39.284787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.284748 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-kube-api-access-r9xc4\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-kube-api-access-r9xc4\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.284987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.284800 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-podinfo\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.284987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.284923 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-certs\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.285250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285217 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-certs" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:39.285250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285231 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-data" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:39.285395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.284952 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-data\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.285395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285303 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-socket\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.285519 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285472 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-envoy\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.285626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285546 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-socket" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:39.285768 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285738 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-token\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-token\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.285889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285859 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-credential-socket\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.285954 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.285902 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2732e9-41b4-4072-a88f-8c02ab3863da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2732e9-41b4-4072-a88f-8c02ab3863da-istiod-ca-cert\") pod \"4a2732e9-41b4-4072-a88f-8c02ab3863da\" (UID: \"4a2732e9-41b4-4072-a88f-8c02ab3863da\") " Nov 03 19:52:39.286596 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.286555 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2732e9-41b4-4072-a88f-8c02ab3863da-istiod-ca-cert" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:52:39.287829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.287796 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a2732e9-41b4-4072-a88f-8c02ab3863da-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.287932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.287835 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.287932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.287850 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.287932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.287864 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.288165 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.288135 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-envoy" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:39.288821 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.288795 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-credential-socket" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:52:39.291983 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.291768 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-kube-api-access-r9xc4" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "kube-api-access-r9xc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:39.294159 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.292571 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-token" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:52:39.294833 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.294793 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-podinfo" pod "4a2732e9-41b4-4072-a88f-8c02ab3863da" (UID: "4a2732e9-41b4-4072-a88f-8c02ab3863da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:52:39.389598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.389492 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.389598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.389526 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.389598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.389540 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9xc4\" (UniqueName: \"kubernetes.io/projected/4a2732e9-41b4-4072-a88f-8c02ab3863da-kube-api-access-r9xc4\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.389598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.389554 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.389598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.389566 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a2732e9-41b4-4072-a88f-8c02ab3863da-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:52:39.604051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.604018 4314 generic.go:356] "Generic (PLEG): container finished" podID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerID="19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b" exitCode=0 Nov 03 19:52:39.604250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.604081 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" event={"ID":"4a2732e9-41b4-4072-a88f-8c02ab3863da","Type":"ContainerDied","Data":"19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b"} Nov 03 19:52:39.604250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.604091 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" Nov 03 19:52:39.604250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.604134 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm" event={"ID":"4a2732e9-41b4-4072-a88f-8c02ab3863da","Type":"ContainerDied","Data":"960b21d960846f538bfbde0c6af55da6c3f897e06cb38bee9b546181908d6597"} Nov 03 19:52:39.604250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.604162 4314 scope.go:122] "RemoveContainer" containerID="19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b" Nov 03 19:52:39.616845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.616825 4314 scope.go:122] "RemoveContainer" containerID="19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b" Nov 03 19:52:39.617157 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:52:39.617132 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b\": container with ID starting with 19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b not found: ID does not exist" containerID="19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b" Nov 03 19:52:39.617223 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.617168 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b"} err="failed to get container status \"19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b\": rpc error: code = NotFound desc = could not find container \"19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b\": container with ID starting with 19997306280da6df8a2c849bb591ce25050bccc9dc3e6cf2b7a221823a45068b not found: ID does not exist" Nov 03 19:52:39.631961 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.631937 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm"] Nov 03 19:52:39.638544 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:39.638520 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-hr4wm"] Nov 03 19:52:40.468808 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:40.468762 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" path="/var/lib/kubelet/pods/4a2732e9-41b4-4072-a88f-8c02ab3863da/volumes" Nov 03 19:52:46.750637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.750604 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv"] Nov 03 19:52:46.751348 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751334 4314 cpu_manager.go:414] "containerMap: removing container" podUID="0b67ac80-3318-49e9-a660-0a4c37f14483" containerName="maas-api" Nov 03 19:52:46.751387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751349 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b67ac80-3318-49e9-a660-0a4c37f14483" containerName="maas-api" Nov 03 19:52:46.751387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751358 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" Nov 03 19:52:46.751387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751362 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" Nov 03 19:52:46.751387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751376 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" Nov 03 19:52:46.751387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751382 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" Nov 03 19:52:46.751542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751398 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" Nov 03 19:52:46.751542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751403 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" Nov 03 19:52:46.751542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751524 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b67ac80-3318-49e9-a660-0a4c37f14483" containerName="maas-api" Nov 03 19:52:46.751542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751534 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" Nov 03 19:52:46.751749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751543 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b551235-3802-4549-9657-8cb3cc22e12b" containerName="istio-proxy" Nov 03 19:52:46.751749 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.751552 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2732e9-41b4-4072-a88f-8c02ab3863da" containerName="istio-proxy" Nov 03 19:52:46.776650 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.756874 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.776650 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.768759 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bdb694-openshift-default-dockercfg-jxjgx\"" Nov 03 19:52:46.776650 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.775305 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv"] Nov 03 19:52:46.828255 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828220 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-envoy\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828369 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gft7\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-kube-api-access-7gft7\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828410 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-credential-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828432 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-data\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828511 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c93810-f8d3-41c6-b220-74c97629b25a-istiod-ca-cert\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828627 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-certs\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828711 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-istio-token\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828857 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828845 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c93810-f8d3-41c6-b220-74c97629b25a-istio-podinfo\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.828898 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.828875 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930172 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930136 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gft7\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-kube-api-access-7gft7\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930181 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-credential-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930206 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-data\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930244 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c93810-f8d3-41c6-b220-74c97629b25a-istiod-ca-cert\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930494 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930464 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-certs\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930591 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930560 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-istio-token\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930622 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-credential-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930775 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-data\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930837 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930796 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c93810-f8d3-41c6-b220-74c97629b25a-istio-podinfo\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930837 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930809 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-certs\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930910 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930847 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.930987 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.930973 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-envoy\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.931052 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.931008 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c93810-f8d3-41c6-b220-74c97629b25a-istiod-ca-cert\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.931187 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.931165 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-socket\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.933032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.933007 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-envoy\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.933150 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.933134 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c93810-f8d3-41c6-b220-74c97629b25a-istio-podinfo\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.938640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.938575 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gft7\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-kube-api-access-7gft7\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:46.938728 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:46.938660 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-istio-token\") pod \"e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:47.093655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:47.092910 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:47.627338 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:47.627303 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv"] Nov 03 19:52:47.628910 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:52:47.628878 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c93810_f8d3_41c6_b220_74c97629b25a.slice/crio-3c78a81572bc0ba4f0f982e32074cb37f79c8b2574ae07937337db03aded2c5d WatchSource:0}: Error finding container 3c78a81572bc0ba4f0f982e32074cb37f79c8b2574ae07937337db03aded2c5d: Status 404 returned error can't find the container with id 3c78a81572bc0ba4f0f982e32074cb37f79c8b2574ae07937337db03aded2c5d Nov 03 19:52:47.631941 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:47.631909 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:47.632037 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:47.631973 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:47.632037 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:47.632003 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:47.750622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:47.750557 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" event={"ID":"99c93810-f8d3-41c6-b220-74c97629b25a","Type":"ContainerStarted","Data":"3c78a81572bc0ba4f0f982e32074cb37f79c8b2574ae07937337db03aded2c5d"} Nov 03 19:52:48.257718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.257680 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7"] Nov 03 19:52:48.263733 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.263701 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.283257 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.283211 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-9f3188-openshift-default-dockercfg-xp46f\"" Nov 03 19:52:48.290041 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.290018 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7"] Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303306 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303474 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303658 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303709 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303821 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7076438-f898-42dc-80dc-9af5392701a5-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303867 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303937 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7076438-f898-42dc-80dc-9af5392701a5-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.303976 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.305453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.304072 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2ng\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-kube-api-access-qs2ng\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.405350 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.405315 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.405529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.405420 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7076438-f898-42dc-80dc-9af5392701a5-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.405529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.405471 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.405652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.405529 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7076438-f898-42dc-80dc-9af5392701a5-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.405767 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.405731 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.406157 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7076438-f898-42dc-80dc-9af5392701a5-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.406946 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.407334 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.407397 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2ng\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-kube-api-access-qs2ng\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.407474 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.407525 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.407717 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.408050 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.408271 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.410772 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.427183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.426138 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7076438-f898-42dc-80dc-9af5392701a5-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.453618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.432181 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2ng\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-kube-api-access-qs2ng\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.453618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.444861 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.609662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.609448 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:48.905349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:48.901698 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" event={"ID":"99c93810-f8d3-41c6-b220-74c97629b25a","Type":"ContainerStarted","Data":"9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040"} Nov 03 19:52:49.106610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.103592 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:49.156031 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.155723 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:49.181602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.178522 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podStartSLOduration=3.178495783 podStartE2EDuration="3.178495783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:48.9295532 +0000 UTC m=+1276.634392542" watchObservedRunningTime="2026-11-03 19:52:49.178495783 +0000 UTC m=+1276.883335126" Nov 03 19:52:49.186351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.186305 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7"] Nov 03 19:52:49.201449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.200815 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:49.201449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.200910 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:49.201449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.200963 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:52:49.912147 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.912107 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" event={"ID":"a7076438-f898-42dc-80dc-9af5392701a5","Type":"ContainerStarted","Data":"5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b"} Nov 03 19:52:49.912551 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.912330 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:49.912551 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.912372 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" event={"ID":"a7076438-f898-42dc-80dc-9af5392701a5","Type":"ContainerStarted","Data":"0674e9c90c23f30496c59ab1384cea7a369cc0975aac0cce64c21e3481b812e2"} Nov 03 19:52:49.913169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.913146 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:52:49.938334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:49.938289 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podStartSLOduration=1.938275139 podStartE2EDuration="1.938275139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:49.932755468 +0000 UTC m=+1277.637595013" watchObservedRunningTime="2026-11-03 19:52:49.938275139 +0000 UTC m=+1277.643114478" Nov 03 19:52:50.610522 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:50.610487 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:50.617039 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:50.617016 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:50.920676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:50.920028 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:50.921652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:50.921624 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 19:52:51.995669 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:52:51.995629 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:52:57.772135 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:52:57.772093 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:53:04.150664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:04.150636 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79"] Nov 03 19:53:04.151073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:04.150829 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" containerID="cri-o://f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e" gracePeriod=30 Nov 03 19:53:04.156811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:04.156782 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk"] Nov 03 19:53:04.157009 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:04.156989 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" containerID="cri-o://bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f" gracePeriod=30 Nov 03 19:53:04.164838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:04.164807 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s"] Nov 03 19:53:04.165423 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:04.165377 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" containerID="cri-o://ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0" gracePeriod=30 Nov 03 19:53:09.291333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.291302 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 19:53:09.291800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.291357 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 19:53:09.690074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.689748 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.776865 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-data\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.776972 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-podinfo\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777014 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-socket\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777154 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-token\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-token\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777364 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-kube-api-access-nf4ks\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-kube-api-access-nf4ks\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777418 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767e45e0-6fdd-4f2e-a884-e335281262ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767e45e0-6fdd-4f2e-a884-e335281262ba-istiod-ca-cert\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777479 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-certs\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777528 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-credential-socket\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777553 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-envoy\") pod \"767e45e0-6fdd-4f2e-a884-e335281262ba\" (UID: \"767e45e0-6fdd-4f2e-a884-e335281262ba\") " Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.777600 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-data" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.778343 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.778672 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-certs" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.778983 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767e45e0-6fdd-4f2e-a884-e335281262ba-istiod-ca-cert" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.779221 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-credential-socket" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.785713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.779390 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-socket" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.805452 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.802263 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-token" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.805452 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.802518 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-kube-api-access-nf4ks" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "kube-api-access-nf4ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.828461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.828062 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:53:09.851363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.850778 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:53:09.859818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.859781 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-envoy" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.863845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.863812 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-podinfo" pod "767e45e0-6fdd-4f2e-a884-e335281262ba" (UID: "767e45e0-6fdd-4f2e-a884-e335281262ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:09.885984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.885952 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-podinfo\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:09.887010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.886988 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-kube-api-access-m8mpl\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-kube-api-access-m8mpl\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887545 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-data\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887628 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-envoy\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887668 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-podinfo\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887711 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-credential-socket\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887839 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-certs\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887886 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54734b03-0f73-4c72-a11f-9af1c765b89b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54734b03-0f73-4c72-a11f-9af1c765b89b-istiod-ca-cert\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887922 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-token\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-token\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.887953 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-credential-socket\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:09.894617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.888008 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-kube-api-access-9l4mk\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-kube-api-access-9l4mk\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:09.902613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.897825 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-data" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.902613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.898537 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54734b03-0f73-4c72-a11f-9af1c765b89b-istiod-ca-cert" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:09.902613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.898932 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-certs" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.902958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.902923 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-credential-socket" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.903412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.903390 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-data\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:09.916336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.904783 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-envoy" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.916336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.905544 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-data" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.916336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.915327 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-credential-socket" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:09.916614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.916358 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-token" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.917201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917160 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.917330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917316 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54734b03-0f73-4c72-a11f-9af1c765b89b-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.917413 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917403 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.917499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917488 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.917635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917573 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4ks\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-kube-api-access-nf4ks\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.917750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917736 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/767e45e0-6fdd-4f2e-a884-e335281262ba-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.917865 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917849 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.917990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917972 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918070 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918160 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918150 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918228 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918315 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918305 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918382 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/767e45e0-6fdd-4f2e-a884-e335281262ba-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918465 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918548 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.918740 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/767e45e0-6fdd-4f2e-a884-e335281262ba-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:09.918917 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.917512 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-kube-api-access-9l4mk" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "kube-api-access-9l4mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.920216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.920186 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-podinfo" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:09.922488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.922458 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-kube-api-access-m8mpl" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "kube-api-access-m8mpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:09.926793 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.926771 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-podinfo" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:09.926982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.926952 4314 generic.go:356] "Generic (PLEG): container finished" podID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerID="bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f" exitCode=0 Nov 03 19:53:09.927164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.927147 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" Nov 03 19:53:09.928471 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.928441 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" event={"ID":"767e45e0-6fdd-4f2e-a884-e335281262ba","Type":"ContainerDied","Data":"bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f"} Nov 03 19:53:09.928573 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.928504 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk" event={"ID":"767e45e0-6fdd-4f2e-a884-e335281262ba","Type":"ContainerDied","Data":"2e62440b58a11f66fc7b5d25b80ec5e5e24c96a9e073d1b0bec756f0b7cf74a2"} Nov 03 19:53:09.928573 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.928537 4314 scope.go:122] "RemoveContainer" containerID="bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f" Nov 03 19:53:09.932888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.932135 4314 generic.go:356] "Generic (PLEG): container finished" podID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerID="ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0" exitCode=0 Nov 03 19:53:09.932888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.932205 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" Nov 03 19:53:09.932888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.932427 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" event={"ID":"54734b03-0f73-4c72-a11f-9af1c765b89b","Type":"ContainerDied","Data":"ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0"} Nov 03 19:53:09.932888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.932474 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s" event={"ID":"54734b03-0f73-4c72-a11f-9af1c765b89b","Type":"ContainerDied","Data":"763c8acf8598b5f3f8f8727116ab05625908725a4d1f0f758c40750e9b9c7b49"} Nov 03 19:53:09.934680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.934566 4314 generic.go:356] "Generic (PLEG): container finished" podID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerID="f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e" exitCode=0 Nov 03 19:53:09.934892 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.934645 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" Nov 03 19:53:09.935195 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.934661 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" event={"ID":"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a","Type":"ContainerDied","Data":"f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e"} Nov 03 19:53:09.935326 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.935310 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" event={"ID":"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a","Type":"ContainerDied","Data":"fe4e0e349c8072473d4b42e5f26479c5c6e041fd757500ed87d4f6b572856098"} Nov 03 19:53:09.943968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.943953 4314 scope.go:122] "RemoveContainer" containerID="bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f" Nov 03 19:53:09.944401 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:09.944376 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f\": container with ID starting with bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f not found: ID does not exist" containerID="bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f" Nov 03 19:53:09.944486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.944406 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f"} err="failed to get container status \"bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f\": rpc error: code = NotFound desc = could not find container \"bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f\": container with ID starting with bf401b5d2cd93bcb0adcfa3ff21694f7c0600cd5f3b9371769591a9b328d2f4f not found: ID does not exist" Nov 03 19:53:09.944486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.944421 4314 scope.go:122] "RemoveContainer" containerID="ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0" Nov 03 19:53:09.961316 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.961291 4314 scope.go:122] "RemoveContainer" containerID="ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0" Nov 03 19:53:09.961644 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:09.961613 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0\": container with ID starting with ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0 not found: ID does not exist" containerID="ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0" Nov 03 19:53:09.961750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.961657 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0"} err="failed to get container status \"ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0\": rpc error: code = NotFound desc = could not find container \"ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0\": container with ID starting with ad6feddd2c9c4c590453a3494976e6131705d7696871949431cc38868b5dcae0 not found: ID does not exist" Nov 03 19:53:09.961750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.961683 4314 scope.go:122] "RemoveContainer" containerID="f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e" Nov 03 19:53:09.978704 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.978616 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk"] Nov 03 19:53:09.979761 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.979744 4314 scope.go:122] "RemoveContainer" containerID="f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e" Nov 03 19:53:09.980228 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:09.980205 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e\": container with ID starting with f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e not found: ID does not exist" containerID="f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e" Nov 03 19:53:09.980365 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.980342 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e"} err="failed to get container status \"f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e\": rpc error: code = NotFound desc = could not find container \"f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e\": container with ID starting with f930ec28ae314785fbb62e477739573b2dbabacceaa3d2aedd14eb2cc7744a9e not found: ID does not exist" Nov 03 19:53:09.985142 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:09.985123 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-qwvqk"] Nov 03 19:53:10.019802 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.019776 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-certs\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:10.019960 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.019842 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istiod-ca-cert\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:10.019960 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.019889 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-token\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-token\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:10.019960 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.019932 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-socket\") pod \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\" (UID: \"19cb32a6-9c8a-4cfd-a80b-73c84f767e4a\") " Nov 03 19:53:10.020122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.020007 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-envoy\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:10.020122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.020043 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-socket\") pod \"54734b03-0f73-4c72-a11f-9af1c765b89b\" (UID: \"54734b03-0f73-4c72-a11f-9af1c765b89b\") " Nov 03 19:53:10.020268 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.020242 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-socket" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:10.020325 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.020273 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istiod-ca-cert" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:10.020422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.020398 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-socket" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:10.020477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.020145 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-certs" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:10.021111 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021086 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.021213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021113 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.021213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021128 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.021213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021141 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.021213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021152 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.021213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021161 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mpl\" (UniqueName: \"kubernetes.io/projected/54734b03-0f73-4c72-a11f-9af1c765b89b-kube-api-access-m8mpl\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.021213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021172 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.021213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.021181 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4mk\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-kube-api-access-9l4mk\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.022084 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.022063 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-envoy" pod "54734b03-0f73-4c72-a11f-9af1c765b89b" (UID: "54734b03-0f73-4c72-a11f-9af1c765b89b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:10.022258 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.022225 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-token" pod "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" (UID: "19cb32a6-9c8a-4cfd-a80b-73c84f767e4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:10.122575 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.122498 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.122575 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.122530 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54734b03-0f73-4c72-a11f-9af1c765b89b-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:10.406685 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.401430 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79"] Nov 03 19:53:10.413169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.413135 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79"] Nov 03 19:53:10.417574 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.417544 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s"] Nov 03 19:53:10.434241 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.434214 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" path="/var/lib/kubelet/pods/19cb32a6-9c8a-4cfd-a80b-73c84f767e4a/volumes" Nov 03 19:53:10.434869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.434851 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" path="/var/lib/kubelet/pods/767e45e0-6fdd-4f2e-a884-e335281262ba/volumes" Nov 03 19:53:10.435219 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.435205 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-hpw5s"] Nov 03 19:53:10.601445 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.601407 4314 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:53:10.601693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:10.601497 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-zkz79" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:53:12.430316 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:12.430282 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" path="/var/lib/kubelet/pods/54734b03-0f73-4c72-a11f-9af1c765b89b/volumes" Nov 03 19:53:21.005225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:21.005130 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb"] Nov 03 19:53:21.005801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:21.005403 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" containerID="cri-o://557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526" gracePeriod=30 Nov 03 19:53:22.244317 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:22.244281 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:53:26.384958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.384921 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:53:26.500603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500261 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-credential-socket\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500325 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istiod-ca-cert\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500411 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-token\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-token\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500558 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-envoy\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500676 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-certs\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500709 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-kube-api-access-mtzhn\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-kube-api-access-mtzhn\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500753 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-socket\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500800 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-data\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.500944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.500837 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-podinfo\") pod \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\" (UID: \"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb\") " Nov 03 19:53:26.501201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.501098 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-credential-socket" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.502169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.501513 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-socket" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.502169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.501774 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.502169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.501798 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.502169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.502056 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-certs" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.502169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.502084 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istiod-ca-cert" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:26.502169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.502146 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-data" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.523631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.517386 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-kube-api-access-mtzhn" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "kube-api-access-mtzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:26.523631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.518427 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-envoy" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:26.584604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.563750 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-podinfo" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:26.584604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.570022 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-token" pod "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" (UID: "32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:26.608646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.605475 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.608646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.605514 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.608646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.605529 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtzhn\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-kube-api-access-mtzhn\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.608646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.605544 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.608646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.605558 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.608646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.605572 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.608646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.605605 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:26.662606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.661872 4314 generic.go:356] "Generic (PLEG): container finished" podID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerID="557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526" exitCode=0 Nov 03 19:53:26.662606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.662068 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" event={"ID":"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb","Type":"ContainerDied","Data":"557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526"} Nov 03 19:53:26.662606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.662121 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" event={"ID":"32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb","Type":"ContainerDied","Data":"bf1308fdef435c77a8213bbcc7c2bf3a5eb3788d41c181e820af09e671d3110f"} Nov 03 19:53:26.662606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.662154 4314 scope.go:122] "RemoveContainer" containerID="557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526" Nov 03 19:53:26.662606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.662329 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb" Nov 03 19:53:26.835607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.829521 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb"] Nov 03 19:53:26.839604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.837519 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b73c810-openshift-default-68d568696f-twzgb"] Nov 03 19:53:26.881676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.881644 4314 scope.go:122] "RemoveContainer" containerID="2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71" Nov 03 19:53:26.902461 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.902353 4314 scope.go:122] "RemoveContainer" containerID="557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526" Nov 03 19:53:26.902849 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:26.902817 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526\": container with ID starting with 557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526 not found: ID does not exist" containerID="557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526" Nov 03 19:53:26.902914 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.902865 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526"} err="failed to get container status \"557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526\": rpc error: code = NotFound desc = could not find container \"557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526\": container with ID starting with 557b6535571117429049ab9c386edf9e03b9bd46f7a169b6d3f86dfdca123526 not found: ID does not exist" Nov 03 19:53:26.902914 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.902891 4314 scope.go:122] "RemoveContainer" containerID="2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71" Nov 03 19:53:26.903190 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:26.903167 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71\": container with ID starting with 2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71 not found: ID does not exist" containerID="2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71" Nov 03 19:53:26.903244 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:26.903201 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71"} err="failed to get container status \"2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71\": rpc error: code = NotFound desc = could not find container \"2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71\": container with ID starting with 2e648b1b5f3c59d73be062c22edc9fb3e1d7a7c8f3e9fbdf54fdced364712a71 not found: ID does not exist" Nov 03 19:53:28.435856 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:28.435809 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" path="/var/lib/kubelet/pods/32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb/volumes" Nov 03 19:53:28.589535 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:28.589504 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2"] Nov 03 19:53:28.590656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:28.590395 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" containerID="cri-o://b51ce5b1de05d0c87a13caef50576f5c15e29409badddeedd2bec40c0976b66a" gracePeriod=30 Nov 03 19:53:32.591943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.591910 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g"] Nov 03 19:53:32.593831 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.593803 4314 cpu_manager.go:414] "containerMap: removing container" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" Nov 03 19:53:32.593982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.593968 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" Nov 03 19:53:32.594105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594086 4314 cpu_manager.go:414] "containerMap: removing container" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" Nov 03 19:53:32.594105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594101 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594123 4314 cpu_manager.go:414] "containerMap: removing container" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594131 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594159 4314 cpu_manager.go:414] "containerMap: removing container" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594167 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594353 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594369 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="54734b03-0f73-4c72-a11f-9af1c765b89b" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594387 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594401 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cb32a6-9c8a-4cfd-a80b-73c84f767e4a" containerName="istio-proxy" Nov 03 19:53:32.594528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.594411 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="767e45e0-6fdd-4f2e-a884-e335281262ba" containerName="istio-proxy" Nov 03 19:53:32.599801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.599774 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.609049 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.609020 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g"] Nov 03 19:53:32.612248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.611250 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-332694-openshift-default-dockercfg-pvj4g\"" Nov 03 19:53:32.691151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.688641 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.695366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.694957 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021031e4-06e6-4109-bea5-44b520187c94-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.712691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.712649 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.713006 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.712962 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.713260 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.713241 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.713610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.713575 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.713800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.713781 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021031e4-06e6-4109-bea5-44b520187c94-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.713997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.713983 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.714092 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.714079 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbf9\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-kube-api-access-sgbf9\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852378 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852463 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021031e4-06e6-4109-bea5-44b520187c94-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852547 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852569 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbf9\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-kube-api-access-sgbf9\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852634 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852698 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021031e4-06e6-4109-bea5-44b520187c94-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852727 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852760 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.852854 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.854363 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.855030 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021031e4-06e6-4109-bea5-44b520187c94-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.855271 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.864487 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.869651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.864777 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.906603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.898898 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021031e4-06e6-4109-bea5-44b520187c94-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.914615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.911971 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.914615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.912362 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.926603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.925613 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbf9\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-kube-api-access-sgbf9\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:32.930603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:32.928287 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:33.060606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.056847 4314 generic.go:356] "Generic (PLEG): container finished" podID="38088147-4b05-4987-82b0-85f536cd2697" containerID="e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697" exitCode=0 Nov 03 19:53:33.060606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.057046 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" event={"ID":"38088147-4b05-4987-82b0-85f536cd2697","Type":"ContainerDied","Data":"e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697"} Nov 03 19:53:33.060606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.057636 4314 scope.go:122] "RemoveContainer" containerID="e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697" Nov 03 19:53:33.060606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.060470 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:33.060606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.060575 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:33.061028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.060635 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:33.384039 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.383935 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g"] Nov 03 19:53:33.395713 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:53:33.392194 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021031e4_06e6_4109_bea5_44b520187c94.slice/crio-9bdb86c1ee96469406ba8484f8f57058c924a9bd39e71b0ca722a47eaca4b737 WatchSource:0}: Error finding container 9bdb86c1ee96469406ba8484f8f57058c924a9bd39e71b0ca722a47eaca4b737: Status 404 returned error can't find the container with id 9bdb86c1ee96469406ba8484f8f57058c924a9bd39e71b0ca722a47eaca4b737 Nov 03 19:53:33.396144 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.396108 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:33.396276 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.396196 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:33.396276 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.396241 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:33.707999 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.707901 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm"] Nov 03 19:53:33.709676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.709649 4314 cpu_manager.go:414] "containerMap: removing container" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" Nov 03 19:53:33.710877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.710857 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ddc1a1-cf76-4b92-aa72-e3b5cbd5c7eb" containerName="istio-proxy" Nov 03 19:53:33.716628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.716608 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.725142 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.723393 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm"] Nov 03 19:53:33.725142 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.723980 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-32a584-openshift-default-dockercfg-tgm45\"" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.832783 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-certs\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.832948 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-socket\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.832995 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-credential-socket\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.833041 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b619441d-0505-47ce-a10c-f24d52df0989-istio-podinfo\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.833121 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b619441d-0505-47ce-a10c-f24d52df0989-istiod-ca-cert\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.833179 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrmm7\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-kube-api-access-vrmm7\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.833256 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-data\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.833370 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-istio-token\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:33.847609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:33.833425 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-envoy\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.011612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.004040 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-data\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.013712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.013669 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-data\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.022606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.014032 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-istio-token\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.179920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.179881 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-envoy\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.183112 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.183076 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-certs\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.184406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.183449 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-socket\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.186563 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.184616 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-credential-socket\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.186834 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.186788 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b619441d-0505-47ce-a10c-f24d52df0989-istio-podinfo\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.187032 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.187016 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b619441d-0505-47ce-a10c-f24d52df0989-istiod-ca-cert\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.187211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.187188 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrmm7\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-kube-api-access-vrmm7\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.198238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.192086 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-credential-socket\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.198238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.192272 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-certs\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.198238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.192441 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-socket\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.198238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.193388 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b619441d-0505-47ce-a10c-f24d52df0989-istiod-ca-cert\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.228643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.227798 4314 generic.go:356] "Generic (PLEG): container finished" podID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerID="b51ce5b1de05d0c87a13caef50576f5c15e29409badddeedd2bec40c0976b66a" exitCode=0 Nov 03 19:53:34.228643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.228035 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" event={"ID":"596a2c8d-09ac-4e12-884b-6b179a74c0b0","Type":"ContainerDied","Data":"b51ce5b1de05d0c87a13caef50576f5c15e29409badddeedd2bec40c0976b66a"} Nov 03 19:53:34.228643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.228120 4314 scope.go:122] "RemoveContainer" containerID="d04dd78fc72ef4351bf080bb4ae3a69e76453616341cc542df87bfa149b09ddf" Nov 03 19:53:34.249792 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.249742 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-envoy\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.252216 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.252173 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrmm7\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-kube-api-access-vrmm7\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.253009 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.252938 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b619441d-0505-47ce-a10c-f24d52df0989-istio-podinfo\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.258963 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.257699 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" event={"ID":"38088147-4b05-4987-82b0-85f536cd2697","Type":"ContainerStarted","Data":"c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503"} Nov 03 19:53:34.292160 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.290888 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" event={"ID":"021031e4-06e6-4109-bea5-44b520187c94","Type":"ContainerStarted","Data":"5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8"} Nov 03 19:53:34.292160 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.290964 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" event={"ID":"021031e4-06e6-4109-bea5-44b520187c94","Type":"ContainerStarted","Data":"9bdb86c1ee96469406ba8484f8f57058c924a9bd39e71b0ca722a47eaca4b737"} Nov 03 19:53:34.317771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.315613 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-istio-token\") pod \"e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.349736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.349477 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:34.408308 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.408249 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podStartSLOduration=2.408229194 podStartE2EDuration="2.408229194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:34.329308974 +0000 UTC m=+1322.034148315" watchObservedRunningTime="2026-11-03 19:53:34.408229194 +0000 UTC m=+1322.113068535" Nov 03 19:53:34.476744 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.476063 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610322 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-socket\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610390 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-credential-socket\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610566 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-kube-api-access-57cml\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-kube-api-access-57cml\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610681 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-podinfo\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610789 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istiod-ca-cert\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610827 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-certs\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610864 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-token\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-token\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.610981 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-data\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.611659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.611085 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-envoy\") pod \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\" (UID: \"596a2c8d-09ac-4e12-884b-6b179a74c0b0\") " Nov 03 19:53:34.613292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.612798 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-credential-socket" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:34.613292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.612966 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-socket" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:34.613292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.613132 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-certs" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:34.613488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.613418 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istiod-ca-cert" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:34.616059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.615725 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-kube-api-access-57cml" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "kube-api-access-57cml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:34.616059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.615992 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-data" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:34.617097 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.617060 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-envoy" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:34.617385 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.617356 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-token" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:34.620011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.619974 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-podinfo" pod "596a2c8d-09ac-4e12-884b-6b179a74c0b0" (UID: "596a2c8d-09ac-4e12-884b-6b179a74c0b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:34.704974 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.702369 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:53:34.704974 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.703253 4314 patch_prober.go:29] interesting pod/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:53:34.704974 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.703307 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:53:34.706064 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:53:34.706034 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb619441d_0505_47ce_a10c_f24d52df0989.slice/crio-7b6a10569fd7e47b06e598647a3f1310e91c732a6cb75697af2b407384e0d535 WatchSource:0}: Error finding container 7b6a10569fd7e47b06e598647a3f1310e91c732a6cb75697af2b407384e0d535: Status 404 returned error can't find the container with id 7b6a10569fd7e47b06e598647a3f1310e91c732a6cb75697af2b407384e0d535 Nov 03 19:53:34.706901 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.706875 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm"] Nov 03 19:53:34.709656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.709575 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:34.710038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.709703 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:34.710038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.709744 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:34.713523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713499 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713527 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713537 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713546 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57cml\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-kube-api-access-57cml\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713556 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713564 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713572 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713597 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.713602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.713606 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596a2c8d-09ac-4e12-884b-6b179a74c0b0-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:34.937547 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.937472 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:34.943371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:34.943342 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:35.409801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.409770 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" Nov 03 19:53:35.410231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.410079 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2" event={"ID":"596a2c8d-09ac-4e12-884b-6b179a74c0b0","Type":"ContainerDied","Data":"0452fcc04fa69877fb444f4e511bf5d739868705f9f59225ca78ab1392d79738"} Nov 03 19:53:35.410231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.410142 4314 scope.go:122] "RemoveContainer" containerID="b51ce5b1de05d0c87a13caef50576f5c15e29409badddeedd2bec40c0976b66a" Nov 03 19:53:35.412066 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.411730 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" event={"ID":"b619441d-0505-47ce-a10c-f24d52df0989","Type":"ContainerStarted","Data":"82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd"} Nov 03 19:53:35.412066 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.411765 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" event={"ID":"b619441d-0505-47ce-a10c-f24d52df0989","Type":"ContainerStarted","Data":"7b6a10569fd7e47b06e598647a3f1310e91c732a6cb75697af2b407384e0d535"} Nov 03 19:53:35.412835 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.412814 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:35.413903 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.413885 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:53:35.456255 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.456206 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podStartSLOduration=2.456191792 podStartE2EDuration="2.456191792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:35.454369222 +0000 UTC m=+1323.159208563" watchObservedRunningTime="2026-11-03 19:53:35.456191792 +0000 UTC m=+1323.161031163" Nov 03 19:53:35.520993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.520858 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2"] Nov 03 19:53:35.529342 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.525465 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a536f35b-openshift-default-c4485d7c-lvbd2"] Nov 03 19:53:35.718657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.718505 4314 patch_prober.go:29] interesting pod/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:53:35.718657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.718594 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:53:35.728721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.728685 4314 scope.go:122] "RemoveContainer" containerID="9d9932792c954ce2e4cb7522336fbe9c3d778f34aaacccd34cf28b11710e7dea" Nov 03 19:53:35.943042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:35.943008 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:53:36.352611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:36.352569 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:36.371615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:36.370881 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:36.512813 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:36.499423 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" path="/var/lib/kubelet/pods/596a2c8d-09ac-4e12-884b-6b179a74c0b0/volumes" Nov 03 19:53:36.512813 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:36.499969 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:36.512813 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:36.500061 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:53:36.705470 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:36.702125 4314 patch_prober.go:29] interesting pod/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:53:36.705470 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:36.702196 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:53:37.713849 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:37.713811 4314 patch_prober.go:29] interesting pod/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:53:37.714481 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:37.714445 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:53:38.750148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:38.744614 4314 patch_prober.go:29] interesting pod/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:53:38.750148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:38.744670 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:53:39.846605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:39.846125 4314 patch_prober.go:29] interesting pod/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:53:39.846605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:39.846184 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:53:40.703417 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:40.703385 4314 patch_prober.go:29] interesting pod/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 19:53:40.703674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:40.703650 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 19:53:41.727150 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:41.727068 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:53:41.729571 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:41.729544 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:53:43.981619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:43.981556 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:53:45.945098 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.945062 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8"] Nov 03 19:53:45.946302 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.946274 4314 cpu_manager.go:414] "containerMap: removing container" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" Nov 03 19:53:45.946302 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.946298 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" Nov 03 19:53:45.946487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.946326 4314 cpu_manager.go:414] "containerMap: removing container" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" Nov 03 19:53:45.946487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.946334 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" Nov 03 19:53:45.946617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.946531 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" Nov 03 19:53:45.946617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.946549 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="596a2c8d-09ac-4e12-884b-6b179a74c0b0" containerName="istio-proxy" Nov 03 19:53:45.954661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.951318 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:45.982900 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.982080 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a99fad-openshift-default-dockercfg-xgm4r\"" Nov 03 19:53:45.987634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:45.984807 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8"] Nov 03 19:53:46.036978 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.036941 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-data\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037021 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037169 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037055 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-podinfo\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037176 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0a4afc-942b-4d68-8eec-86290a693e1a-istiod-ca-cert\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037267 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-certs\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037439 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037353 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-envoy\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037439 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037401 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-token\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037454 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zzb\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-kube-api-access-p9zzb\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.037614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.037558 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-credential-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139596 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-data\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139684 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139710 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-podinfo\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139752 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0a4afc-942b-4d68-8eec-86290a693e1a-istiod-ca-cert\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139814 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-certs\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139851 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-envoy\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139897 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-token\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.139956 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zzb\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-kube-api-access-p9zzb\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.140096 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-credential-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.140698 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-credential-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.140985 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-data\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.143602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.141201 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.159608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.156455 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0a4afc-942b-4d68-8eec-86290a693e1a-istiod-ca-cert\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.159608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.157293 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-certs\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.208665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.201419 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zzb\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-kube-api-access-p9zzb\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.220974 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.220921 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-podinfo\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.221651 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.221625 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-envoy\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.269197 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.269163 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-token\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.304736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.304704 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:46.744225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.744193 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8"] Nov 03 19:53:46.747288 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:53:46.747257 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0a4afc_942b_4d68_8eec_86290a693e1a.slice/crio-0eef96424a8f34ebe086f0bc3dee942c469dcff8d987613af4691822a0ad7de9 WatchSource:0}: Error finding container 0eef96424a8f34ebe086f0bc3dee942c469dcff8d987613af4691822a0ad7de9: Status 404 returned error can't find the container with id 0eef96424a8f34ebe086f0bc3dee942c469dcff8d987613af4691822a0ad7de9 Nov 03 19:53:46.751410 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.751372 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:46.751492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.751462 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:46.751528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:46.751506 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:47.201533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:47.185875 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" event={"ID":"db0a4afc-942b-4d68-8eec-86290a693e1a","Type":"ContainerStarted","Data":"47a961f8e87b41f26476d018eefa8846721d4001cee96b54b4c2af802e36f601"} Nov 03 19:53:47.201533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:47.185946 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" event={"ID":"db0a4afc-942b-4d68-8eec-86290a693e1a","Type":"ContainerStarted","Data":"0eef96424a8f34ebe086f0bc3dee942c469dcff8d987613af4691822a0ad7de9"} Nov 03 19:53:47.241304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:47.227660 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podStartSLOduration=2.227641356 podStartE2EDuration="2.227641356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:47.227131868 +0000 UTC m=+1334.931971253" watchObservedRunningTime="2026-11-03 19:53:47.227641356 +0000 UTC m=+1334.932480698" Nov 03 19:53:47.307608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:47.307051 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:47.307608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:47.307566 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:53:47.307950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:47.307650 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:53:48.313778 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:48.313739 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:48.543984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:48.543947 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95"] Nov 03 19:53:48.544162 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:48.544140 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" containerID="cri-o://b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8" gracePeriod=30 Nov 03 19:53:48.548412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:48.548378 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv"] Nov 03 19:53:48.548626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:48.548600 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" containerID="cri-o://ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2" gracePeriod=30 Nov 03 19:53:49.212685 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:49.212653 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:49.213617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:49.213557 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:53:52.019068 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:52.019027 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:53:53.884404 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:53.884379 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:53:54.027080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.026252 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-certs\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.027080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.026377 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-kube-api-access-fmrkn\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-kube-api-access-fmrkn\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.030042 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.029990 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-certs" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.031446 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a0624f-b804-49d4-9e59-d696bd226f1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a0624f-b804-49d4-9e59-d696bd226f1c-istiod-ca-cert\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.031517 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-token\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-token\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.031641 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-credential-socket\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.031792 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-socket\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.031869 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-envoy\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.031997 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-data\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.032062 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-podinfo\") pod \"a4a0624f-b804-49d4-9e59-d696bd226f1c\" (UID: \"a4a0624f-b804-49d4-9e59-d696bd226f1c\") " Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.033136 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.040847 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-token" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.041083 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-credential-socket" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.041461 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-socket" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.050574 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-data" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.054529 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.053792 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a0624f-b804-49d4-9e59-d696bd226f1c-istiod-ca-cert" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:54.068404 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.063492 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-podinfo" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:54.068404 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.066697 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-envoy" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.078337 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.078057 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-kube-api-access-fmrkn" pod "a4a0624f-b804-49d4-9e59-d696bd226f1c" (UID: "a4a0624f-b804-49d4-9e59-d696bd226f1c"). InnerVolumeSpecName "kube-api-access-fmrkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:54.135739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135660 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.135739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135699 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.135739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135715 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.135739 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135730 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.136085 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135744 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.136085 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135762 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrkn\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-kube-api-access-fmrkn\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.136085 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135778 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a0624f-b804-49d4-9e59-d696bd226f1c-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.136085 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.135792 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a0624f-b804-49d4-9e59-d696bd226f1c-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.290867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.290833 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:53:54.335299 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.335259 4314 generic.go:356] "Generic (PLEG): container finished" podID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerID="b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8" exitCode=0 Nov 03 19:53:54.335486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.335349 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" Nov 03 19:53:54.335486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.335443 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" event={"ID":"a4a0624f-b804-49d4-9e59-d696bd226f1c","Type":"ContainerDied","Data":"b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8"} Nov 03 19:53:54.335637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.335486 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95" event={"ID":"a4a0624f-b804-49d4-9e59-d696bd226f1c","Type":"ContainerDied","Data":"b97970cb1cbf83d0f4caca04688f40f26c708ab3af32d394c52bc2dea5f79c9e"} Nov 03 19:53:54.335637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.335511 4314 scope.go:122] "RemoveContainer" containerID="b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8" Nov 03 19:53:54.344010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.343841 4314 generic.go:356] "Generic (PLEG): container finished" podID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerID="ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2" exitCode=0 Nov 03 19:53:54.344010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.343988 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" event={"ID":"98ea0ebd-b1b0-4075-863b-0794864e0c24","Type":"ContainerDied","Data":"ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2"} Nov 03 19:53:54.344203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.344026 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" event={"ID":"98ea0ebd-b1b0-4075-863b-0794864e0c24","Type":"ContainerDied","Data":"bb5ee4e3c87f3dba43942eb69d5cca966c647bd23108c1fc322379c21f71b9dd"} Nov 03 19:53:54.344203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.344080 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv" Nov 03 19:53:54.356453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.356433 4314 scope.go:122] "RemoveContainer" containerID="b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8" Nov 03 19:53:54.357188 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:54.357037 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8\": container with ID starting with b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8 not found: ID does not exist" containerID="b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8" Nov 03 19:53:54.357188 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.357075 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8"} err="failed to get container status \"b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8\": rpc error: code = NotFound desc = could not find container \"b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8\": container with ID starting with b3d8f6fa3e92357bd4eda443c8ad4d0df9060dee904ee562edb7408ed82653b8 not found: ID does not exist" Nov 03 19:53:54.357188 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.357099 4314 scope.go:122] "RemoveContainer" containerID="ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2" Nov 03 19:53:54.383640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.374314 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95"] Nov 03 19:53:54.383640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.382146 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-66t95"] Nov 03 19:53:54.401455 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.401426 4314 scope.go:122] "RemoveContainer" containerID="ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2" Nov 03 19:53:54.402206 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:53:54.402176 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2\": container with ID starting with ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2 not found: ID does not exist" containerID="ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2" Nov 03 19:53:54.402326 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.402217 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2"} err="failed to get container status \"ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2\": rpc error: code = NotFound desc = could not find container \"ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2\": container with ID starting with ef9eef84af9ba551d71c812856ab74cfed5a4d804990e83950f63ba71fcc56b2 not found: ID does not exist" Nov 03 19:53:54.441196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.441166 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-data\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.441457 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.441439 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-kube-api-access-qnbdd\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-kube-api-access-qnbdd\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.441611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.441574 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-envoy\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.441772 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.441746 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-credential-socket\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.441967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.441954 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-token\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-token\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.442051 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.442042 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-certs\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.442200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.442191 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ea0ebd-b1b0-4075-863b-0794864e0c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ea0ebd-b1b0-4075-863b-0794864e0c24-istiod-ca-cert\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.442275 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.442267 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-socket\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.442430 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.442419 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-podinfo\") pod \"98ea0ebd-b1b0-4075-863b-0794864e0c24\" (UID: \"98ea0ebd-b1b0-4075-863b-0794864e0c24\") " Nov 03 19:53:54.446041 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.446019 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-podinfo" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:53:54.446135 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.446059 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-credential-socket" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.446269 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.446248 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-data" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.448217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.448040 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-certs" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.448217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.448058 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-socket" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.449000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.447987 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ea0ebd-b1b0-4075-863b-0794864e0c24-istiod-ca-cert" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:53:54.449945 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.449912 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-envoy" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:53:54.455495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.455473 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-kube-api-access-qnbdd" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "kube-api-access-qnbdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:54.457997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.457968 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" path="/var/lib/kubelet/pods/a4a0624f-b804-49d4-9e59-d696bd226f1c/volumes" Nov 03 19:53:54.468100 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.468024 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-token" pod "98ea0ebd-b1b0-4075-863b-0794864e0c24" (UID: "98ea0ebd-b1b0-4075-863b-0794864e0c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:53:54.544468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544432 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544466 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544476 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnbdd\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-kube-api-access-qnbdd\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544487 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544496 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544504 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ea0ebd-b1b0-4075-863b-0794864e0c24-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544512 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544522 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ea0ebd-b1b0-4075-863b-0794864e0c24-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.544661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.544531 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ea0ebd-b1b0-4075-863b-0794864e0c24-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:53:54.715950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.715918 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv"] Nov 03 19:53:54.722933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:54.722899 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-877bv"] Nov 03 19:53:55.365283 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.364597 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th"] Nov 03 19:53:55.365797 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.365773 4314 cpu_manager.go:414] "containerMap: removing container" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" Nov 03 19:53:55.365891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.365797 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" Nov 03 19:53:55.365891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.365819 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" Nov 03 19:53:55.365891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.365827 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" Nov 03 19:53:55.367292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.367266 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" containerName="istio-proxy" Nov 03 19:53:55.367292 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.367292 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a0624f-b804-49d4-9e59-d696bd226f1c" containerName="istio-proxy" Nov 03 19:53:55.397605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.396794 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th"] Nov 03 19:53:55.397605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.396913 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.400718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.400696 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-332694-serving-cert\"" Nov 03 19:53:55.584409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.584367 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq2cf\" (UniqueName: \"kubernetes.io/projected/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-kube-api-access-jq2cf\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.584703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.584675 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.584832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.584814 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.584942 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.584926 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-metrics-certs\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.687029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.686142 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-metrics-certs\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.687029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.686223 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq2cf\" (UniqueName: \"kubernetes.io/projected/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-kube-api-access-jq2cf\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.687029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.686391 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.687029 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.686455 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.687350 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.687070 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.691631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.690945 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-metrics-certs\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.693335 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.693314 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.697485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.697109 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq2cf\" (UniqueName: \"kubernetes.io/projected/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-kube-api-access-jq2cf\") pod \"maas-api-e2e-inf-b-332694-d77564d9b-9n9th\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:55.718737 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:55.718713 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:56.048604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:56.037938 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th"] Nov 03 19:53:56.118143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:56.101002 4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:53:56.835182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:56.835137 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ea0ebd-b1b0-4075-863b-0794864e0c24" path="/var/lib/kubelet/pods/98ea0ebd-b1b0-4075-863b-0794864e0c24/volumes" Nov 03 19:53:56.857608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:56.843536 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" event={"ID":"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3","Type":"ContainerStarted","Data":"cb7ea86f636b357859d72b01253adde652f566f1d8e1dbceabed94eb4673b1d0"} Nov 03 19:53:57.740994 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:57.740960 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" event={"ID":"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3","Type":"ContainerStarted","Data":"775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb"} Nov 03 19:53:57.743030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:57.741096 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:53:57.765231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:57.765176 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" podStartSLOduration=1.836137626 podStartE2EDuration="2.765162643s" totalImagesPullingTime="929.025017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:57.761212456 +0000 UTC m=+1345.466051800" watchObservedRunningTime="2026-11-03 19:53:57.765162643 +0000 UTC m=+1345.470001982" Nov 03 19:53:58.846149 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:58.845331 4314 generic.go:356] "Generic (PLEG): container finished" podID="99c93810-f8d3-41c6-b220-74c97629b25a" containerID="9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040" exitCode=0 Nov 03 19:53:58.847080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:58.847047 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" event={"ID":"99c93810-f8d3-41c6-b220-74c97629b25a","Type":"ContainerDied","Data":"9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040"} Nov 03 19:53:58.847694 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:58.847674 4314 scope.go:122] "RemoveContainer" containerID="9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040" Nov 03 19:53:58.851701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:58.851657 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:58.851931 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:58.851908 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:58.852108 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:58.852071 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:53:59.855946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:53:59.855912 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" event={"ID":"99c93810-f8d3-41c6-b220-74c97629b25a","Type":"ContainerStarted","Data":"4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953"} Nov 03 19:54:00.095129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:00.094546 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:54:00.095129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:00.094628 4314 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:54:00.095129 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:00.094698 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 19:54:01.095776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:01.095732 4314 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:54:01.096266 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:01.095808 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 19:54:02.094117 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:02.094080 4314 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:54:02.094392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:02.094360 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 19:54:03.093832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:03.093793 4314 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:54:03.094400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:03.094364 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 19:54:03.614688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:03.614642 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46"] Nov 03 19:54:03.615946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:03.614886 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" containerID="cri-o://c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503" gracePeriod=30 Nov 03 19:54:04.054971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:04.054938 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 19:54:04.120877 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:04.120817 4314 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:54:04.121540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:04.121506 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 19:54:04.912898 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:04.912867 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:54:05.094503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:05.094461 4314 patch_prober.go:29] interesting pod/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 19:54:05.094503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:05.094528 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 19:54:06.099731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:06.099698 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:54:06.100929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:06.100906 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 19:54:08.941790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.941763 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.972481 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-istio-token\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-istio-token\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.972599 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-data\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.972700 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-certs\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.972854 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-credential-socket\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.972911 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-kube-api-access-795z9\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.973014 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38088147-4b05-4987-82b0-85f536cd2697-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38088147-4b05-4987-82b0-85f536cd2697-istiod-ca-cert\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.973054 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-envoy\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.973108 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38088147-4b05-4987-82b0-85f536cd2697-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38088147-4b05-4987-82b0-85f536cd2697-istio-podinfo\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.973248 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-socket\") pod \"38088147-4b05-4987-82b0-85f536cd2697\" (UID: \"38088147-4b05-4987-82b0-85f536cd2697\") " Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.974443 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-credential-socket" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.974713 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-socket" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.975411 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-certs" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.975598 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38088147-4b05-4987-82b0-85f536cd2697-istiod-ca-cert" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:08.977636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.975653 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-data" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:08.978467 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.978002 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-kube-api-access-795z9" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "kube-api-access-795z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:08.983865 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.979611 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-envoy" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:08.983865 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.979942 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-istio-token" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:08.984176 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:08.984147 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38088147-4b05-4987-82b0-85f536cd2697-istio-podinfo" pod "38088147-4b05-4987-82b0-85f536cd2697" (UID: "38088147-4b05-4987-82b0-85f536cd2697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:54:09.076806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076771 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076807 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-795z9\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-kube-api-access-795z9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076824 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38088147-4b05-4987-82b0-85f536cd2697-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076838 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076852 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38088147-4b05-4987-82b0-85f536cd2697-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076868 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076881 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38088147-4b05-4987-82b0-85f536cd2697-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076898 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.076996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.076912 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38088147-4b05-4987-82b0-85f536cd2697-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:09.296489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.296457 4314 generic.go:356] "Generic (PLEG): container finished" podID="38088147-4b05-4987-82b0-85f536cd2697" containerID="c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503" exitCode=0 Nov 03 19:54:09.296734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.296529 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" Nov 03 19:54:09.296734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.296545 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" event={"ID":"38088147-4b05-4987-82b0-85f536cd2697","Type":"ContainerDied","Data":"c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503"} Nov 03 19:54:09.296734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.296648 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46" event={"ID":"38088147-4b05-4987-82b0-85f536cd2697","Type":"ContainerDied","Data":"d5c752167b24d93cb3a89b324fe319ed7e053d5c506b015c2303cb173aa16388"} Nov 03 19:54:09.296734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.296685 4314 scope.go:122] "RemoveContainer" containerID="c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503" Nov 03 19:54:09.308968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.308950 4314 scope.go:122] "RemoveContainer" containerID="e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697" Nov 03 19:54:09.320643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.320594 4314 scope.go:122] "RemoveContainer" containerID="c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503" Nov 03 19:54:09.320888 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:54:09.320866 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503\": container with ID starting with c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503 not found: ID does not exist" containerID="c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503" Nov 03 19:54:09.320954 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.320894 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503"} err="failed to get container status \"c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503\": rpc error: code = NotFound desc = could not find container \"c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503\": container with ID starting with c9e2ec08fd98e28d0481b9441b430d28b1d5263a309c0bcde834adc0d1ba3503 not found: ID does not exist" Nov 03 19:54:09.320954 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.320913 4314 scope.go:122] "RemoveContainer" containerID="e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697" Nov 03 19:54:09.321171 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:54:09.321153 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697\": container with ID starting with e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697 not found: ID does not exist" containerID="e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697" Nov 03 19:54:09.321232 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.321175 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697"} err="failed to get container status \"e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697\": rpc error: code = NotFound desc = could not find container \"e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697\": container with ID starting with e7e21b2a48e6707de893490ea3fa4705296db30bdd9747c1768180524ad6d697 not found: ID does not exist" Nov 03 19:54:09.327617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.327595 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46"] Nov 03 19:54:09.332547 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:09.332526 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9f84de-openshift-default-6954b8b997-qkj46"] Nov 03 19:54:10.203405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.203371 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr"] Nov 03 19:54:10.204232 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.204214 4314 cpu_manager.go:414] "containerMap: removing container" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" Nov 03 19:54:10.204305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.204233 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" Nov 03 19:54:10.204431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.204420 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" Nov 03 19:54:10.204468 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.204437 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" Nov 03 19:54:10.209714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.209689 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.217771 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.217726 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a99fad-serving-cert\"" Nov 03 19:54:10.218887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.218502 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr"] Nov 03 19:54:10.298743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.298703 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-metrics-certs\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.298931 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.298760 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.299038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.299016 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcs59\" (UniqueName: \"kubernetes.io/projected/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-kube-api-access-zcs59\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.299114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.299095 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-maas-api-tls\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.400710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.400672 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-maas-api-tls\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.400887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.400869 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-metrics-certs\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.400966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.400900 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.401050 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.401028 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcs59\" (UniqueName: \"kubernetes.io/projected/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-kube-api-access-zcs59\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.401637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.401614 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.403795 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.403775 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-maas-api-tls\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.404440 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.404419 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-metrics-certs\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.409643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.409621 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcs59\" (UniqueName: \"kubernetes.io/projected/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-kube-api-access-zcs59\") pod \"maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.497959 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.497834 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38088147-4b05-4987-82b0-85f536cd2697" path="/var/lib/kubelet/pods/38088147-4b05-4987-82b0-85f536cd2697/volumes" Nov 03 19:54:10.535269 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.535228 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:10.935137 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:10.935070 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr"] Nov 03 19:54:11.355351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:11.355304 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" event={"ID":"b7b22c85-eb65-412d-8cc8-48dd9e92aecb","Type":"ContainerStarted","Data":"a88030d833700933dc1a86ea5242805465a37daff4967dd972026666c044b3cb"} Nov 03 19:54:12.385518 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:12.385482 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" event={"ID":"b7b22c85-eb65-412d-8cc8-48dd9e92aecb","Type":"ContainerStarted","Data":"b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80"} Nov 03 19:54:12.385958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:12.385620 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:12.406312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:12.406261 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" podStartSLOduration=1.815880763 podStartE2EDuration="2.40624864s" totalImagesPullingTime="590.367877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:12.402691617 +0000 UTC m=+1360.107530957" watchObservedRunningTime="2026-11-03 19:54:12.40624864 +0000 UTC m=+1360.111088000" Nov 03 19:54:13.222755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.222659 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7"] Nov 03 19:54:13.288604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.224195 4314 cpu_manager.go:414] "containerMap: removing container" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" Nov 03 19:54:13.289867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.289824 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38088147-4b05-4987-82b0-85f536cd2697" containerName="istio-proxy" Nov 03 19:54:13.531607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.531011 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7"] Nov 03 19:54:13.545748 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.532120 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.556944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.556921 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0d0987-openshift-default-dockercfg-q5nng\"" Nov 03 19:54:13.668566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.668525 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-socket\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.684719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.670368 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istiod-ca-cert\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.684719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.684212 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cd78\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-kube-api-access-5cd78\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.684719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.684377 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-podinfo\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.684719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.684550 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-token\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.685078 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.684699 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-envoy\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.685078 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.684891 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-certs\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.685078 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.684931 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-credential-socket\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.685224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.685099 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-data\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.786769 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.786677 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-data\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.786769 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.786768 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-socket\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.786787 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istiod-ca-cert\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.786832 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cd78\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-kube-api-access-5cd78\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.786924 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-podinfo\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.786970 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-token\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787037 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-envoy\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787084 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-certs\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787120 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-credential-socket\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787143 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-data\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787182 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-socket\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787521 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-certs\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787648 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-credential-socket\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.787900 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.787881 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istiod-ca-cert\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.789661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.789627 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-envoy\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.790134 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.790111 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-podinfo\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.795276 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.795251 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cd78\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-kube-api-access-5cd78\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.795621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.795600 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-token\") pod \"e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:13.876331 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:13.876304 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:14.216738 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.216706 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7"] Nov 03 19:54:14.217701 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:54:14.217676 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda422e62b_e92c_4dff_8ef6_5722d98b7e06.slice/crio-53153730c72c6922c3b4f223f1440932248b8fda651a0c975df90a1653fa6c71 WatchSource:0}: Error finding container 53153730c72c6922c3b4f223f1440932248b8fda651a0c975df90a1653fa6c71: Status 404 returned error can't find the container with id 53153730c72c6922c3b4f223f1440932248b8fda651a0c975df90a1653fa6c71 Nov 03 19:54:14.221353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.221312 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:14.221443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.221406 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:14.221549 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.221458 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:14.520968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.520933 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" event={"ID":"a422e62b-e92c-4dff-8ef6-5722d98b7e06","Type":"ContainerStarted","Data":"89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f"} Nov 03 19:54:14.521121 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.520990 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" event={"ID":"a422e62b-e92c-4dff-8ef6-5722d98b7e06","Type":"ContainerStarted","Data":"53153730c72c6922c3b4f223f1440932248b8fda651a0c975df90a1653fa6c71"} Nov 03 19:54:14.541054 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.540687 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podStartSLOduration=1.540668457 podStartE2EDuration="1.540668457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:14.539378681 +0000 UTC m=+1362.244218034" watchObservedRunningTime="2026-11-03 19:54:14.540668457 +0000 UTC m=+1362.245507798" Nov 03 19:54:14.880658 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:14.879730 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:15.883422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:15.883389 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:16.579719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:16.579678 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:16.580713 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:16.580692 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:54:18.555149 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:18.550740 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 19:54:22.062572 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:54:22.062529 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:54:25.032699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.032661 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm"] Nov 03 19:54:25.056714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.056552 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.072266 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.072231 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-225668-openshift-default-dockercfg-xmp27\"" Nov 03 19:54:25.086890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.086858 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm"] Nov 03 19:54:25.200400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.200363 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.200641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.200419 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.200740 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.200652 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.200740 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.200698 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.200827 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.200792 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503264b1-ec82-44d1-ab06-e466c4c9827e-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.200975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.200959 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.201022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.201008 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.201126 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.201106 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzps\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-kube-api-access-4hzps\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.201168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.201152 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.313888 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.313951 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.313997 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503264b1-ec82-44d1-ab06-e466c4c9827e-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.314153 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.314219 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.314268 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzps\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-kube-api-access-4hzps\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.314311 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.314434 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.314661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.314480 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.342682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.315406 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.342682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.317433 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.342682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.317741 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.342682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.318016 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.342682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.318115 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503264b1-ec82-44d1-ab06-e466c4c9827e-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.342682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.332129 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.343353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.343331 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.357619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.350462 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.367984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.366234 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzps\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-kube-api-access-4hzps\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-wnbmm\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.410990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.410954 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:25.907182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.907151 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm"] Nov 03 19:54:25.908235 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:54:25.908207 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503264b1_ec82_44d1_ab06_e466c4c9827e.slice/crio-d98599fbafc83627a2ebe9c623a06b2aa3264a06a6841b8442ff5aadef4a80a3 WatchSource:0}: Error finding container d98599fbafc83627a2ebe9c623a06b2aa3264a06a6841b8442ff5aadef4a80a3: Status 404 returned error can't find the container with id d98599fbafc83627a2ebe9c623a06b2aa3264a06a6841b8442ff5aadef4a80a3 Nov 03 19:54:25.911285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.911249 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:25.911375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.911326 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:25.911375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:25.911360 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:26.701608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:26.701195 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" event={"ID":"503264b1-ec82-44d1-ab06-e466c4c9827e","Type":"ContainerStarted","Data":"763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca"} Nov 03 19:54:26.701608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:26.701257 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" event={"ID":"503264b1-ec82-44d1-ab06-e466c4c9827e","Type":"ContainerStarted","Data":"d98599fbafc83627a2ebe9c623a06b2aa3264a06a6841b8442ff5aadef4a80a3"} Nov 03 19:54:26.725511 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:26.725458 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podStartSLOduration=1.725445685 podStartE2EDuration="1.725445685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:26.721457407 +0000 UTC m=+1374.426296749" watchObservedRunningTime="2026-11-03 19:54:26.725445685 +0000 UTC m=+1374.430285025" Nov 03 19:54:27.093528 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:27.093490 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp"] Nov 03 19:54:27.093790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:27.093746 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" containerID="cri-o://40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75" gracePeriod=30 Nov 03 19:54:27.411923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:27.411829 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:27.418496 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:27.418466 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:27.715708 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:27.715631 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:27.716669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:27.716644 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:54:31.021659 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:31.012149 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:54:32.815952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.815926 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:54:32.929605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.902192 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-data\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.930910 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.930863 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-socket\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.931308 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.931288 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-certs\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.935786 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.935763 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-envoy\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.942516 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.936016 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-token\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.957628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.953204 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-credential-socket\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.962698 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.962664 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-kube-api-access-5452c\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-kube-api-access-5452c\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.962937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.962917 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istiod-ca-cert\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.963131 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.963115 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-podinfo\") pod \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\" (UID: \"9dfd69b0-5b9c-4e8e-81ae-700825b0b858\") " Nov 03 19:54:32.986602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.935665 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-certs" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:32.986977 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.986945 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istiod-ca-cert" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:54:32.987295 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.944797 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-data" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:32.987418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:32.952660 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-socket" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:33.013180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.013143 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-credential-socket" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:33.013653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.013628 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-envoy" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:54:33.049915 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.039118 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-token" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:33.080604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.074846 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-kube-api-access-5452c" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "kube-api-access-5452c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116848 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116885 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116900 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116914 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116929 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5452c\" (UniqueName: \"kubernetes.io/projected/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-kube-api-access-5452c\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116946 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116959 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.118670 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.116975 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.182501 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.182444 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-podinfo" pod "9dfd69b0-5b9c-4e8e-81ae-700825b0b858" (UID: "9dfd69b0-5b9c-4e8e-81ae-700825b0b858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:54:33.266358 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.266325 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dfd69b0-5b9c-4e8e-81ae-700825b0b858-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:54:33.297034 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.296981 4314 generic.go:356] "Generic (PLEG): container finished" podID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerID="40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75" exitCode=0 Nov 03 19:54:33.297550 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.297522 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" event={"ID":"9dfd69b0-5b9c-4e8e-81ae-700825b0b858","Type":"ContainerDied","Data":"40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75"} Nov 03 19:54:33.297732 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.297710 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" event={"ID":"9dfd69b0-5b9c-4e8e-81ae-700825b0b858","Type":"ContainerDied","Data":"9a353f47e03e1c65ed77e6d371445e5ea3fb80945559431357a42c80cd392e2c"} Nov 03 19:54:33.297869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.297856 4314 scope.go:122] "RemoveContainer" containerID="40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75" Nov 03 19:54:33.298205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.298191 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp" Nov 03 19:54:33.507789 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.501110 4314 scope.go:122] "RemoveContainer" containerID="40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75" Nov 03 19:54:33.546787 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:54:33.546749 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75\": container with ID starting with 40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75 not found: ID does not exist" containerID="40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75" Nov 03 19:54:33.547329 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.547020 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75"} err="failed to get container status \"40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75\": rpc error: code = NotFound desc = could not find container \"40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75\": container with ID starting with 40d5ede3417a44b5576eb30ec836979f12d9a02f07b5717563867a2427e85c75 not found: ID does not exist" Nov 03 19:54:33.667947 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.648530 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp"] Nov 03 19:54:33.667947 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:33.651063 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-cktrp"] Nov 03 19:54:34.317444 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:34.317388 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:34.345944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:34.345900 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:34.391757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:34.391207 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:34.537538 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:34.534717 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:54:34.619194 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:34.619109 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" path="/var/lib/kubelet/pods/9dfd69b0-5b9c-4e8e-81ae-700825b0b858/volumes" Nov 03 19:54:34.656611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:34.655685 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:34.656611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:34.656125 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:54:35.328821 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:35.328772 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 19:54:35.350793 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:35.350751 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:54:35.351634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:35.351608 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:54:35.408095 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:35.408054 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:35.471707 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:35.427475 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:54:35.682661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:35.656739 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:35.682661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:35.669772 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:54:36.056621 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:36.055787 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:54:36.380595 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:36.362815 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:36.412735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:36.412692 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:36.517968 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:36.515913 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:54:36.549648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:36.545245 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:36.580076 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:36.580034 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:36.659727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:36.659674 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:37.082634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.069822 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:37.082634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.071432 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:54:37.382878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.382773 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:37.463649 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.425509 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:37.570647 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.560961 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:37.625622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.611856 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:54:37.625622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.613281 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:54:37.669139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.669006 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:54:37.851838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.668951 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:37.856775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.856739 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:54:38.036206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.036178 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:54:38.038316 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.038205 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:54:38.039080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:37.963036 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Nov 03 19:54:38.073016 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.072951 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:54:38.384458 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.383604 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:38.428451 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.428406 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:38.612777 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.612724 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:38.614694 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.614653 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:38.958606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:38.936677 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:54:39.000465 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.000415 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:39.024606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.003504 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:39.151950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.151892 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 19:54:39.182827 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.182792 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:54:39.311662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.303786 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:39.458493 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.458462 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:54:39.471597 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.460253 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:54:39.527340 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.511639 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:54:39.527340 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.511741 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:54:39.527340 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.511774 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:54:39.628681 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.628443 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:54:39.639616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.637734 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:54:39.648788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:39.648764 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:54:40.036003 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:40.035954 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:40.095629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:40.092959 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:54:40.146788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:40.146354 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:54:47.806431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.805935 4314 generic.go:356] "Generic (PLEG): container finished" podID="b619441d-0505-47ce-a10c-f24d52df0989" containerID="82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd" exitCode=0 Nov 03 19:54:47.806431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.806087 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" event={"ID":"b619441d-0505-47ce-a10c-f24d52df0989","Type":"ContainerDied","Data":"82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd"} Nov 03 19:54:47.807084 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.806949 4314 scope.go:122] "RemoveContainer" containerID="82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd" Nov 03 19:54:47.810196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.810157 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:47.810391 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.810371 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:47.810523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.810506 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:47.821764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.819143 4314 generic.go:356] "Generic (PLEG): container finished" podID="021031e4-06e6-4109-bea5-44b520187c94" containerID="5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8" exitCode=0 Nov 03 19:54:47.821764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.819274 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz"] Nov 03 19:54:47.821764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.820510 4314 cpu_manager.go:414] "containerMap: removing container" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" Nov 03 19:54:47.821764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.820531 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" Nov 03 19:54:47.821764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.820738 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfd69b0-5b9c-4e8e-81ae-700825b0b858" containerName="istio-proxy" Nov 03 19:54:47.827838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.826546 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" event={"ID":"021031e4-06e6-4109-bea5-44b520187c94","Type":"ContainerDied","Data":"5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8"} Nov 03 19:54:47.827838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.827058 4314 scope.go:122] "RemoveContainer" containerID="5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8" Nov 03 19:54:47.828166 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.828136 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.830414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.830384 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz"] Nov 03 19:54:47.850688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.843110 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:47.850688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.844110 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:47.850688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.844179 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:47.935706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.935352 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.935706 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.935395 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.935929 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.935904 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.935997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.935949 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.935997 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.935976 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.936107 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.936003 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cw9m\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-kube-api-access-5cw9m\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.936260 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.936231 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.936321 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.936287 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e1c8f5-9908-4067-818e-98a7a90eebbf-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:47.936745 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:47.936723 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.038947 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039185 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039220 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039255 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039283 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039308 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039337 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cw9m\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-kube-api-access-5cw9m\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039462 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.039498 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e1c8f5-9908-4067-818e-98a7a90eebbf-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.040446 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e1c8f5-9908-4067-818e-98a7a90eebbf-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.041672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.041087 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.042380 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.041773 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.042380 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.041868 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.042380 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.042003 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.046637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.044889 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.046637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.045562 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.051305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.051279 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.053264 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.053238 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cw9m\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-kube-api-access-5cw9m\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.167406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.166942 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:48.478507 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.478461 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:48.478644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.478556 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:48.478644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.478616 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:48.481808 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.481784 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz"] Nov 03 19:54:48.829780 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.829730 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" event={"ID":"b619441d-0505-47ce-a10c-f24d52df0989","Type":"ContainerStarted","Data":"ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac"} Nov 03 19:54:48.831515 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.831473 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" event={"ID":"98e1c8f5-9908-4067-818e-98a7a90eebbf","Type":"ContainerStarted","Data":"6f4d0dbe69a2d97b10958bcce8dd8ee4fa2a8ad8654cba4b5277030ec2e28583"} Nov 03 19:54:48.831652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.831541 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" event={"ID":"98e1c8f5-9908-4067-818e-98a7a90eebbf","Type":"ContainerStarted","Data":"83c408f43e6f78c6df728068643cbedd3671f738885beb3872dbcb3723859b93"} Nov 03 19:54:48.833934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.833908 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" event={"ID":"021031e4-06e6-4109-bea5-44b520187c94","Type":"ContainerStarted","Data":"91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862"} Nov 03 19:54:48.909511 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.909454 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podStartSLOduration=1.909434796 podStartE2EDuration="1.909434796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:48.896072273 +0000 UTC m=+1396.600911614" watchObservedRunningTime="2026-11-03 19:54:48.909434796 +0000 UTC m=+1396.614274136" Nov 03 19:54:48.937816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.937783 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:54:48.939057 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.939027 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:48.939215 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:48.939091 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:49.173995 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.173900 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:49.194284 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.194145 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:49.194759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.194551 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:49.352001 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.351965 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:54:49.353224 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.353195 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:54:49.353328 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.353256 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 19:54:49.946607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.946129 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:49.946607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:49.946201 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:50.171606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:50.170411 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:50.171606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:50.170470 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:50.355607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:50.355342 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:54:50.355607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:50.355445 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 19:54:50.553607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:50.552631 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:54:50.946605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:50.943464 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:50.946605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:50.943525 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:51.174662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:51.172995 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:51.174662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:51.173059 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:51.355185 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:51.354795 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:54:51.355185 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:51.354859 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 19:54:51.454609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:51.454002 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:54:51.938604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:51.937455 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:51.938604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:51.937517 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:52.171127 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:54:52.155736 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:54:52.173187 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:52.173151 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:52.173551 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:52.173484 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:52.355067 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:52.353565 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:54:52.355067 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:52.354917 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 19:54:52.940828 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:52.938541 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:52.940828 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:52.938623 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:53.171575 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:53.171535 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:53.172622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:53.172184 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:53.402603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:53.401869 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:54:53.402603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:53.401931 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 19:54:53.971604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:53.958485 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:53.971604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:53.958548 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:54.253017 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.252982 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:54.278709 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.278665 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:54.330915 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.330814 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.367439 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.367403 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:54:54.367747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.367715 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 19:54:54.390897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.390834 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.404849 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.404774 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.433496 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.433453 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:54:54.507115 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.507012 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.623190 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.623054 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.650606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.649308 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:54:54.686092 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.686053 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.726604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.725834 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.918664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.892235 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:54.945340 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.945301 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:54.945663 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:54.945629 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:55.245664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.242988 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:55.245664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.243053 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:55.439662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.327941 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:55.439662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.367661 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 19:54:55.439662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.367778 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 19:54:55.439662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.420734 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:55.439662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.421272 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:54:55.595654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.516109 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:55.595654 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.595200 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:54:55.596200 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.596176 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp"] Nov 03 19:54:55.632025 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.631978 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:54:55.658660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.655869 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:55.691727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.691684 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:55.741619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.735678 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:54:55.977696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.972984 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 03 19:54:55.977696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:55.973055 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 03 19:54:56.162990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.162955 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:54:56.163199 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.163185 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp"] Nov 03 19:54:56.167616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.163325 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:54:56.193230 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.193201 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:54:56.193491 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.193477 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:54:56.193642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.193630 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:54:56.193795 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.193783 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:54:56.194680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.180746 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 19:54:56.194927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.194880 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 19:54:56.195509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.177136 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.221637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.216160 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:54:56.222294 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.222252 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-11c1a8-openshift-default-dockercfg-7l2db\"" Nov 03 19:54:56.274615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.265459 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:54:56.355225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.348412 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.368947 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369112 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369254 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369321 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x56b\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-kube-api-access-5x56b\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369513 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369672 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369749 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369824 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.371553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.369874 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.433662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.431201 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:54:56.479869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.479828 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.480498 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.480473 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.480682 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.480666 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.518856 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.518826 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.519309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.519282 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.519601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.519559 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.570656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.489074 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.574700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.486935 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.517018 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.575488 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.575531 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.519395 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.564333 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.572937 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.561751 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.563645 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.575894 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.576248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.575980 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x56b\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-kube-api-access-5x56b\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.601111 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.578800 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.683302 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.683270 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:54:56.713392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.713361 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:54:56.747678 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.745856 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 19:54:56.751868 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.751826 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.791537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.777806 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x56b\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-kube-api-access-5x56b\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:56.948817 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.948731 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:54:56.951219 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.951192 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 19:54:56.996233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:56.996201 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:57.356401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.356374 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:57.380206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.370990 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:57.401479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.401437 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 19:54:57.516195 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.516158 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp"] Nov 03 19:54:57.517206 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:54:57.517180 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c349a0a_69ad_4f14_ba0d_292c4a009b17.slice/crio-aa72097a62248796282cebcc178a6a71cf21dad9e07c387ca0d60d36b04e8c0e WatchSource:0}: Error finding container aa72097a62248796282cebcc178a6a71cf21dad9e07c387ca0d60d36b04e8c0e: Status 404 returned error can't find the container with id aa72097a62248796282cebcc178a6a71cf21dad9e07c387ca0d60d36b04e8c0e Nov 03 19:54:57.520297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.520266 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:57.520363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.520339 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:57.520401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.520374 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:54:57.920928 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.920540 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" event={"ID":"3c349a0a-69ad-4f14-ba0d-292c4a009b17","Type":"ContainerStarted","Data":"1c30e50cbf659931eb96c80e8fbfee4c08d3e6197971195b936edc7c2940643c"} Nov 03 19:54:57.921553 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.921368 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" event={"ID":"3c349a0a-69ad-4f14-ba0d-292c4a009b17","Type":"ContainerStarted","Data":"aa72097a62248796282cebcc178a6a71cf21dad9e07c387ca0d60d36b04e8c0e"} Nov 03 19:54:57.950211 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.950149 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podStartSLOduration=2.950133032 podStartE2EDuration="2.950133032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:57.948559918 +0000 UTC m=+1405.653399259" watchObservedRunningTime="2026-11-03 19:54:57.950133032 +0000 UTC m=+1405.654972371" Nov 03 19:54:57.996991 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:57.996953 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:59.003610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:59.003565 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:59.978460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:59.977163 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:54:59.990631 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:54:59.988451 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:55:01.539998 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.539960 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk"] Nov 03 19:55:01.558153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.558119 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:01.564510 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.564470 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-11c1a8-dockercfg-qmc6n\"" Nov 03 19:55:01.586008 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.585977 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk"] Nov 03 19:55:01.588022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.587983 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-11c1a8\"" Nov 03 19:55:01.700608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.684404 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b72a3f1-73e6-471a-987f-fb30f6b73752-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:01.701891 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.701854 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6ns\" (UniqueName: \"kubernetes.io/projected/7b72a3f1-73e6-471a-987f-fb30f6b73752-kube-api-access-tv6ns\") pod \"payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:01.815679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.803877 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6ns\" (UniqueName: \"kubernetes.io/projected/7b72a3f1-73e6-471a-987f-fb30f6b73752-kube-api-access-tv6ns\") pod \"payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:01.815679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.804043 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b72a3f1-73e6-471a-987f-fb30f6b73752-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:01.815679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.805107 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b72a3f1-73e6-471a-987f-fb30f6b73752-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:01.846573 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.846489 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6ns\" (UniqueName: \"kubernetes.io/projected/7b72a3f1-73e6-471a-987f-fb30f6b73752-kube-api-access-tv6ns\") pod \"payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:01.888152 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:01.887888 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:02.363676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:02.352850 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk"] Nov 03 19:55:03.025390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:03.025352 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" event={"ID":"7b72a3f1-73e6-471a-987f-fb30f6b73752","Type":"ContainerStarted","Data":"2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa"} Nov 03 19:55:03.025863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:03.025411 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" event={"ID":"7b72a3f1-73e6-471a-987f-fb30f6b73752","Type":"ContainerStarted","Data":"aa68efc1078081c9d8e29fc0ed5aa757dbcf0d90ee7572126574555223bc97db"} Nov 03 19:55:03.025863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:03.025434 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:03.048226 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:03.048173 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" podStartSLOduration=2.048160512 podStartE2EDuration="2.048160512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:03.04294074 +0000 UTC m=+1410.747780081" watchObservedRunningTime="2026-11-03 19:55:03.048160512 +0000 UTC m=+1410.752999852" Nov 03 19:55:14.052975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:14.052937 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 19:55:21.998374 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:55:21.998330 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:55:22.624141 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.624106 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7"] Nov 03 19:55:22.631015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.630989 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.640149 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.640123 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7a62d4-openshift-default-dockercfg-vb879\"" Nov 03 19:55:22.649555 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.649530 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7"] Nov 03 19:55:22.709300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.709261 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5z2s\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-kube-api-access-q5z2s\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.709687 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.709658 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2d0438-5595-42d4-821e-5618f05382fd-istio-podinfo\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.709921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.709880 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.712614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.712575 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-istio-token\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.712942 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.712886 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-envoy\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.713153 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.713134 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-data\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.713280 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.713264 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-credential-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.713414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.713399 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-certs\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.713600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.713563 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2d0438-5595-42d4-821e-5618f05382fd-istiod-ca-cert\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823103 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2d0438-5595-42d4-821e-5618f05382fd-istio-podinfo\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823202 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823272 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-istio-token\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823427 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-envoy\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823512 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-data\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823544 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-credential-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823618 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-certs\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.826948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823696 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2d0438-5595-42d4-821e-5618f05382fd-istiod-ca-cert\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.855630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.831838 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.855630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.840280 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-credential-socket\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.855630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.840527 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-data\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.855630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.840803 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-certs\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.855630 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.841417 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2d0438-5595-42d4-821e-5618f05382fd-istiod-ca-cert\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.928731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.823724 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5z2s\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-kube-api-access-q5z2s\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:22.972366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:22.955737 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-istio-token\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:23.043620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.015302 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5z2s\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-kube-api-access-q5z2s\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:23.043620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.021500 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2d0438-5595-42d4-821e-5618f05382fd-istio-podinfo\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:23.076609 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.072936 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-envoy\") pod \"e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:23.253127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.253094 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:23.597678 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:55:23.597553 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2d0438_5595_42d4_821e_5618f05382fd.slice/crio-40b6cd60b61036c4b6f56fc4cf450b0f78cb388f4ffb45696a9e7f0cc90b399f WatchSource:0}: Error finding container 40b6cd60b61036c4b6f56fc4cf450b0f78cb388f4ffb45696a9e7f0cc90b399f: Status 404 returned error can't find the container with id 40b6cd60b61036c4b6f56fc4cf450b0f78cb388f4ffb45696a9e7f0cc90b399f Nov 03 19:55:23.598412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.598386 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7"] Nov 03 19:55:23.619611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.601133 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:23.619611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.601220 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:23.619611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:23.601261 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:24.073606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:24.069825 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:55:24.590604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:24.589878 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" event={"ID":"fa2d0438-5595-42d4-821e-5618f05382fd","Type":"ContainerStarted","Data":"14de153be85fc5e6443a99cc5621f5e19a9867e9f647f414a2b55368494e3463"} Nov 03 19:55:24.590604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:24.589954 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" event={"ID":"fa2d0438-5595-42d4-821e-5618f05382fd","Type":"ContainerStarted","Data":"40b6cd60b61036c4b6f56fc4cf450b0f78cb388f4ffb45696a9e7f0cc90b399f"} Nov 03 19:55:24.619632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:24.619493 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podStartSLOduration=2.619475473 podStartE2EDuration="2.619475473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:24.614360874 +0000 UTC m=+1432.319200217" watchObservedRunningTime="2026-11-03 19:55:24.619475473 +0000 UTC m=+1432.324314815" Nov 03 19:55:25.263606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:25.256964 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:25.846602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:25.843946 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:26.489602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:26.488683 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:26.836138 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:26.836048 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:26.867607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:26.857822 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:55:27.481055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:27.440744 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:27.481055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:27.443796 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:27.481055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:27.463955 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:27.612633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:27.491034 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:27.612633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:27.534740 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:55:27.682608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:27.614632 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:28.248606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.247878 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:28.262474 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.262431 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:28.455595 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.454695 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:55:28.472381 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.472336 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:28.490917 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.490888 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:55:28.491537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.491518 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:55:28.502697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.491431 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:28.646694 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.637357 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:28.646694 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:28.637876 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:55:29.336617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.335267 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:55:29.341326 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.340765 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:29.341326 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.341272 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:55:29.349349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.349315 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:29.352247 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.352207 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 19:55:29.376605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.354471 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:55:29.376605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.373491 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:29.376605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.373704 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:55:29.495718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.495670 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:55:29.544607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.501691 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:55:29.855605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:29.848683 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:30.386625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.376326 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:30.418108 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.388025 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:55:30.418108 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.392310 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:55:30.418108 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.404749 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:30.505833 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.504811 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:30.515523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.506214 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:55:30.515523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.507322 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:55:30.866399 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:30.866364 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:55:31.355693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.347685 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:55:31.380607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.373354 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 19:55:31.380607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.373598 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:31.425605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.398048 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:31.425605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.415432 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 03 19:55:31.425605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.415510 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:55:31.561608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.553203 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:55:31.847603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:31.844659 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:32.381608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.379671 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:32.381608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.380284 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:55:32.407022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.406972 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:32.455608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.435700 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:32.542737 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.542691 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:32.635660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.568503 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:32.730617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.598270 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:55:32.849043 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.849002 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:32.949776 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:32.850019 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:55:33.227387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.225209 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr"] Nov 03 19:55:33.335604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.334882 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr"] Nov 03 19:55:33.335604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.334984 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:55:33.335604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.335025 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:55:33.335604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.335060 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:55:33.354861 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.352701 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.412602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.409695 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:33.442942 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.442661 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:33.650486 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.650446 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.650926 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.650896 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.651421 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.651392 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.784955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.784919 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t429v\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-kube-api-access-t429v\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.785235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.785212 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.785518 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.785380 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.786218 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.786131 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.786387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.786370 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.786515 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.786501 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.787557 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.783230 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:33.796668 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.796645 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:55:33.920878 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.920784 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:55:33.931449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.931403 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.931732 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.931708 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.933203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.933172 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.933796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.933747 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t429v\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-kube-api-access-t429v\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.934011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.933979 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.934251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.934189 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.934745 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.934512 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.934897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.934878 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.939648 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.935683 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.945895 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.945494 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:33.951889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.951855 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:55:33.985307 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.977207 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:33.985307 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.978190 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:55:33.998787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:33.998760 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.149560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.149520 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:34.151910 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.151616 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.152056 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.152010 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.165231 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.165202 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.166045 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.166023 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.210722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.210551 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.211673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.211632 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.549889 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.549498 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t429v\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-kube-api-access-t429v\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-wztdr\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.555672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.555642 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:55:34.555800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.555706 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:55:34.585680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.573599 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:34.585680 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.573814 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:34.645895 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.645651 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:34.694602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.688540 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:55:34.694602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.688662 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:55:34.727323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:34.725547 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:55:35.251811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.199184 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr"] Nov 03 19:55:35.319109 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:55:35.305755 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bdfc8b_2b4e_4646_8381_7166c60090b9.slice/crio-f0f1b65e21c35b243865d8210f13e05823bbc99e6e6a67d880b210568c8d7eaf WatchSource:0}: Error finding container f0f1b65e21c35b243865d8210f13e05823bbc99e6e6a67d880b210568c8d7eaf: Status 404 returned error can't find the container with id f0f1b65e21c35b243865d8210f13e05823bbc99e6e6a67d880b210568c8d7eaf Nov 03 19:55:35.329090 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.329039 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:35.329235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.329144 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:35.329235 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.329191 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:35.499600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.496740 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:55:35.499600 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.496803 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:35.642201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.641668 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" event={"ID":"e7bdfc8b-2b4e-4646-8381-7166c60090b9","Type":"ContainerStarted","Data":"50f8b5c2ad68c01035dabea00a0d84f0ec2201b677a2eee0a26e437a51f81b51"} Nov 03 19:55:35.642201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.641736 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" event={"ID":"e7bdfc8b-2b4e-4646-8381-7166c60090b9","Type":"ContainerStarted","Data":"f0f1b65e21c35b243865d8210f13e05823bbc99e6e6a67d880b210568c8d7eaf"} Nov 03 19:55:35.647570 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:35.647544 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:36.649449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:36.649406 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:55:36.650000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:36.649471 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 19:55:37.648798 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:37.648762 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:55:37.648979 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:37.648813 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 19:55:38.648759 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:38.648712 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:55:38.649261 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:38.648772 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 19:55:39.648625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:39.648071 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:55:39.648625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:39.648147 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 19:55:40.648899 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:40.648693 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:55:40.648899 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:40.648760 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 19:55:41.648234 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:41.648191 4314 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 19:55:41.648489 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:41.648265 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 19:55:42.674983 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:42.672914 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:42.674983 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:42.674769 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:42.678724 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:42.678664 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 19:55:42.698623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:42.696356 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podStartSLOduration=10.696310774 podStartE2EDuration="10.696310774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:35.675709474 +0000 UTC m=+1443.380548815" watchObservedRunningTime="2026-11-03 19:55:42.696310774 +0000 UTC m=+1450.401150149" Nov 03 19:55:43.757816 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:43.757775 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:55:43.776400 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:43.757864 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:44.712380 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.712252 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl"] Nov 03 19:55:44.722268 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.722240 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl"] Nov 03 19:55:44.722637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.722558 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.736732 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.736706 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3cee6a-openshift-default-dockercfg-zhq4h\"" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942294 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-certs\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942465 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-envoy\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942516 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-socket\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942618 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzzpt\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-kube-api-access-qzzpt\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942718 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-token\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942784 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c314-2fcc-4302-a6ac-11834c80cf9a-istiod-ca-cert\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942869 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-podinfo\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942920 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-data\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:44.998832 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:44.942968 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-credential-socket\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059378 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-token\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059464 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c314-2fcc-4302-a6ac-11834c80cf9a-istiod-ca-cert\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059540 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-podinfo\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059606 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-data\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059655 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-credential-socket\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059749 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-certs\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059885 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-envoy\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.059930 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-socket\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.060009 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzzpt\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-kube-api-access-qzzpt\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.060894 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-credential-socket\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.062611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.061602 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c314-2fcc-4302-a6ac-11834c80cf9a-istiod-ca-cert\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.071533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.067626 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-certs\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.071533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.067888 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-data\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.099707 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.074886 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-socket\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.117780 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.117735 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-envoy\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.126603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.118571 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzzpt\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-kube-api-access-qzzpt\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.131625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.130537 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-podinfo\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.131625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.131363 4314 generic.go:356] "Generic (PLEG): container finished" podID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerID="763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca" exitCode=0 Nov 03 19:55:45.131625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.131516 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" event={"ID":"503264b1-ec82-44d1-ab06-e466c4c9827e","Type":"ContainerDied","Data":"763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca"} Nov 03 19:55:45.132213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.132174 4314 scope.go:122] "RemoveContainer" containerID="763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca" Nov 03 19:55:45.145638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.145133 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-token\") pod \"e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.185348 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.185823 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.185890 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.186401 4314 generic.go:356] "Generic (PLEG): container finished" podID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerID="47a961f8e87b41f26476d018eefa8846721d4001cee96b54b4c2af802e36f601" exitCode=0 Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.186507 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" event={"ID":"db0a4afc-942b-4d68-8eec-86290a693e1a","Type":"ContainerDied","Data":"47a961f8e87b41f26476d018eefa8846721d4001cee96b54b4c2af802e36f601"} Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.187085 4314 scope.go:122] "RemoveContainer" containerID="47a961f8e87b41f26476d018eefa8846721d4001cee96b54b4c2af802e36f601" Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.193993 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.194122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.194082 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.196463 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.194457 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.231046 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.229901 4314 generic.go:356] "Generic (PLEG): container finished" podID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerID="89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f" exitCode=0 Nov 03 19:55:45.231046 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.229965 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" event={"ID":"a422e62b-e92c-4dff-8ef6-5722d98b7e06","Type":"ContainerDied","Data":"89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f"} Nov 03 19:55:45.231975 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.231750 4314 scope.go:122] "RemoveContainer" containerID="89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f" Nov 03 19:55:45.299911 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.299860 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.300066 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.300044 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.300272 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.300145 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.351057 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.350356 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:45.760741 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.760712 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl"] Nov 03 19:55:45.773168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.772254 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.773168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.772373 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:45.773168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:45.772420 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:46.330634 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.318275 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" event={"ID":"503264b1-ec82-44d1-ab06-e466c4c9827e","Type":"ContainerStarted","Data":"487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7"} Nov 03 19:55:46.339605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.336306 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" event={"ID":"db0a4afc-942b-4d68-8eec-86290a693e1a","Type":"ContainerStarted","Data":"5fe2a850e7bfa3446eb624f0f7f06efda8dc8ddc0c72f0f38a9e08111dbdfb17"} Nov 03 19:55:46.377604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.377339 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" event={"ID":"c956c314-2fcc-4302-a6ac-11834c80cf9a","Type":"ContainerStarted","Data":"e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47"} Nov 03 19:55:46.377604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.377410 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" event={"ID":"c956c314-2fcc-4302-a6ac-11834c80cf9a","Type":"ContainerStarted","Data":"c94b469a076bcf8d0534ed44416ece5a094571fa9aa6eb83f00f4946afb60d6f"} Nov 03 19:55:46.412333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.411956 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:55:46.412333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.412068 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:46.412333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.412110 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:46.472357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.472311 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" event={"ID":"a422e62b-e92c-4dff-8ef6-5722d98b7e06","Type":"ContainerStarted","Data":"75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9"} Nov 03 19:55:46.516445 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.516382 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podStartSLOduration=2.516363276 podStartE2EDuration="2.516363276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:46.410620323 +0000 UTC m=+1454.115459664" watchObservedRunningTime="2026-11-03 19:55:46.516363276 +0000 UTC m=+1454.221202619" Nov 03 19:55:46.582898 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.582717 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:55:46.884285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.884188 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:46.884552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.884508 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:46.884978 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:46.884078 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:55:47.319608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.312661 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:55:47.354632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.344387 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:47.354632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.354508 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:47.379322 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.354375 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:47.403950 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.403914 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:47.539615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.412006 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:47.539615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.537816 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:47.683846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.682296 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:47.794604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.793880 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:55:47.899606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.887011 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:47.899606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:47.887075 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:48.308607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.308448 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:48.308607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.308525 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:48.416612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.415619 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:48.416612 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.416062 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:55:48.417175 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.417072 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:48.417175 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.417119 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:48.522624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.509303 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:55:48.570657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.564860 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:55:48.626864 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.615344 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:48.949665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.926016 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:48.949665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:48.926086 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:49.022673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.014812 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:49.066157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.066074 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:49.246455 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.246423 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:55:49.307093 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.307012 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:49.307301 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.307099 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:49.420640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.413867 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:49.420640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.413927 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:49.577669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.573657 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:49.577669 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.574240 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:55:49.780748 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.779530 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:55:49.930408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.920934 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:49.930408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:49.921012 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:50.042607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.016851 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:55:50.139282 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.069096 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:55:50.497638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.484051 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:50.497638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.484124 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:50.497638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.486232 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:55:50.528862 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.499470 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:50.673812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.673768 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:50.697182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.697140 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:50.898703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.896741 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:50.898703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:50.896814 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:51.027729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.027682 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.054601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.031202 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.311607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.307984 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:51.311607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.308060 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:51.385619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.371276 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.419031 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.418992 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:51.419325 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.419295 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:51.486499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.486458 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.487902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.487870 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.488697 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.488676 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:55:51.501006 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.500969 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:55:51.503445 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.503422 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:55:51.587639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.587529 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.610725 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.610680 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.682333 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.682284 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:51.881414 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.881317 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:51.881679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:51.881652 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:52.032727 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.032676 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:52.035080 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.035052 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:55:52.082854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.059729 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:52.199705 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.196634 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm"] Nov 03 19:55:52.279412 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.279380 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.329764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.329714 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:52.345626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.330018 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:52.368492 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.367592 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm"] Nov 03 19:55:52.402632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.402512 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac9f8e4-f04d-48af-8f72-d364a9402172-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.402632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.402627 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.402913 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.402716 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.402913 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.402801 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.403047 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.403016 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4h4n\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-kube-api-access-h4h4n\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.403146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.403102 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.403217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.403147 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.403217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.403184 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.437725 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.403350 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.480143 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.469163 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f58171-openshift-default-dockercfg-kj42z\"" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.498028 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519141 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4h4n\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-kube-api-access-h4h4n\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519219 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519256 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519290 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519412 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519467 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac9f8e4-f04d-48af-8f72-d364a9402172-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519549 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519653 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.519736 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.520280 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.560382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.517747 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:55:52.662405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.628686 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4h4n\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-kube-api-access-h4h4n\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.662405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.628799 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:55:52.662405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.637191 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.697606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.672419 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.769598 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.693258 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.782608 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:55:52.770566 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:55:52.782608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.696053 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:55:52.782608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.733842 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:52.782608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.770907 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:52.782608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.734447 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac9f8e4-f04d-48af-8f72-d364a9402172-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.783432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.693492 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:52.900428 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.896196 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:55:53.032457 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:52.913225 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:53.032457 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.023161 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:53.032457 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.024562 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:53.032457 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.031527 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:53.033226 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.033151 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:53.033377 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.033363 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:55:53.267189 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.267132 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:53.355655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.311460 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:53.594044 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.463412 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:53.594633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.477894 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:53.594811 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.594770 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:53.594925 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.522763 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:55:53.596601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.506931 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:55:53.597196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.477958 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:53.597365 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.597342 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:53.597921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.569339 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:55:53.598366 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.580729 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:53.626533 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.603238 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:55:54.007623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:53.999995 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:55:54.007623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.003741 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:55:54.019602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.018410 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:55:54.044606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.039693 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:54.044606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.042422 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:54.314279 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.314236 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:54.317314 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.317281 4314 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 03 19:55:54.317570 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.317541 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 03 19:55:54.472442 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.472407 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:55:54.472752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.472722 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:55:54.562957 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.562927 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:55:54.579330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.574257 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:55:54.606015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.599572 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:54.762607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.694757 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:55:54.776755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.776556 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:55:54.855671 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.836212 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:55:54.994790 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.988975 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:55:54.998130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:54.998103 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:55:55.071686 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.058114 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm"] Nov 03 19:55:55.071686 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:55:55.060693 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac9f8e4_f04d_48af_8f72_d364a9402172.slice/crio-b90d1f8d69ef0d176da27aca38f84fe75318afd4c58ede2934b00f75669fdc4a WatchSource:0}: Error finding container b90d1f8d69ef0d176da27aca38f84fe75318afd4c58ede2934b00f75669fdc4a: Status 404 returned error can't find the container with id b90d1f8d69ef0d176da27aca38f84fe75318afd4c58ede2934b00f75669fdc4a Nov 03 19:55:55.142887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.141639 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:55.142887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.141741 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:55.142887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.141803 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:55:55.313563 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.313530 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:55:55.314906 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.314886 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 19:55:55.433557 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.433470 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:55:55.436125 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.436102 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:55:55.442902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.442879 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 19:55:55.762336 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.762280 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" event={"ID":"3ac9f8e4-f04d-48af-8f72-d364a9402172","Type":"ContainerStarted","Data":"2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202"} Nov 03 19:55:55.762747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.762358 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" event={"ID":"3ac9f8e4-f04d-48af-8f72-d364a9402172","Type":"ContainerStarted","Data":"b90d1f8d69ef0d176da27aca38f84fe75318afd4c58ede2934b00f75669fdc4a"} Nov 03 19:55:55.787557 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:55.787498 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podStartSLOduration=3.78748132 podStartE2EDuration="3.78748132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:55.781645843 +0000 UTC m=+1463.486485186" watchObservedRunningTime="2026-11-03 19:55:55.78748132 +0000 UTC m=+1463.492320669" Nov 03 19:55:56.317406 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:56.317372 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:56.323168 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:56.323137 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:56.773848 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:56.773794 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:55:56.774624 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:55:56.773882 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:56:00.232652 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.196380 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z"] Nov 03 19:56:00.499868 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.499838 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:00.509541 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.509515 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f58171\"" Nov 03 19:56:00.509923 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.509904 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f58171-dockercfg-q5pmq\"" Nov 03 19:56:00.544571 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.544540 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z"] Nov 03 19:56:00.581083 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.581048 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76cdce8-912a-419d-adb9-d2b6e8749cd3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:00.581242 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.581219 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9tn\" (UniqueName: \"kubernetes.io/projected/a76cdce8-912a-419d-adb9-d2b6e8749cd3-kube-api-access-8s9tn\") pod \"payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:00.683635 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.683598 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76cdce8-912a-419d-adb9-d2b6e8749cd3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:00.683807 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.683725 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9tn\" (UniqueName: \"kubernetes.io/projected/a76cdce8-912a-419d-adb9-d2b6e8749cd3-kube-api-access-8s9tn\") pod \"payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:00.684409 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.684391 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76cdce8-912a-419d-adb9-d2b6e8749cd3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:00.692453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.692430 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9tn\" (UniqueName: \"kubernetes.io/projected/a76cdce8-912a-419d-adb9-d2b6e8749cd3-kube-api-access-8s9tn\") pod \"payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:00.850136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:00.850046 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:01.486924 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:01.486888 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z"] Nov 03 19:56:01.487496 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:56:01.487467 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76cdce8_912a_419d_adb9_d2b6e8749cd3.slice/crio-91e8e77a53d2037d7e8427399fdf2bd3e5f5d1c1c8a7ab3560fbc13b7750e63e WatchSource:0}: Error finding container 91e8e77a53d2037d7e8427399fdf2bd3e5f5d1c1c8a7ab3560fbc13b7750e63e: Status 404 returned error can't find the container with id 91e8e77a53d2037d7e8427399fdf2bd3e5f5d1c1c8a7ab3560fbc13b7750e63e Nov 03 19:56:02.098157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:02.098118 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" event={"ID":"a76cdce8-912a-419d-adb9-d2b6e8749cd3","Type":"ContainerStarted","Data":"3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687"} Nov 03 19:56:02.098157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:02.098188 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" event={"ID":"a76cdce8-912a-419d-adb9-d2b6e8749cd3","Type":"ContainerStarted","Data":"91e8e77a53d2037d7e8427399fdf2bd3e5f5d1c1c8a7ab3560fbc13b7750e63e"} Nov 03 19:56:02.098536 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:02.098414 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:02.116149 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:02.116091 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" podStartSLOduration=2.11607608 podStartE2EDuration="2.11607608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:02.114237218 +0000 UTC m=+1469.819076558" watchObservedRunningTime="2026-11-03 19:56:02.11607608 +0000 UTC m=+1469.820915419" Nov 03 19:56:13.125377 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:13.125340 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 19:56:20.339632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.339386 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr"] Nov 03 19:56:20.376555 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.375129 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr"] Nov 03 19:56:20.376555 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.375254 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.382637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.382611 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:56:20.391490 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.384925 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 19:56:20.501033 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.500993 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.501220 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.501138 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.501543 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.501516 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rkn\" (UniqueName: \"kubernetes.io/projected/abfa53fa-0668-4bb0-9b23-25a41519d799-kube-api-access-s9rkn\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.603971 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.603870 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rkn\" (UniqueName: \"kubernetes.io/projected/abfa53fa-0668-4bb0-9b23-25a41519d799-kube-api-access-s9rkn\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.604375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.604348 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.604534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.604432 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.605449 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.605422 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.607729 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.607691 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.612015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.611992 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rkn\" (UniqueName: \"kubernetes.io/projected/abfa53fa-0668-4bb0-9b23-25a41519d799-kube-api-access-s9rkn\") pod \"maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:20.711617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:20.708839 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:21.814017 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:21.813986 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr"] Nov 03 19:56:22.001439 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:56:22.001401 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:56:22.594854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:22.594799 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" event={"ID":"abfa53fa-0668-4bb0-9b23-25a41519d799","Type":"ContainerStarted","Data":"7f7e994fbb42714ed8b8df2e7bf23646e5b818981af4d1244914e98a30f9a781"} Nov 03 19:56:22.595059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:22.594885 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" event={"ID":"abfa53fa-0668-4bb0-9b23-25a41519d799","Type":"ContainerStarted","Data":"e6151d230bdd22842ffa5147bef4d9321234462e170fad4b2f4f0fb49aa3d5ff"} Nov 03 19:56:22.616628 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:22.614128 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" podStartSLOduration=2.614110746 podStartE2EDuration="2.614110746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:22.61381868 +0000 UTC m=+1490.318658020" watchObservedRunningTime="2026-11-03 19:56:22.614110746 +0000 UTC m=+1490.318950086" Nov 03 19:56:23.668604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:23.663328 4314 generic.go:356] "Generic (PLEG): container finished" podID="abfa53fa-0668-4bb0-9b23-25a41519d799" containerID="7f7e994fbb42714ed8b8df2e7bf23646e5b818981af4d1244914e98a30f9a781" exitCode=0 Nov 03 19:56:23.668604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:23.663572 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" event={"ID":"abfa53fa-0668-4bb0-9b23-25a41519d799","Type":"ContainerDied","Data":"7f7e994fbb42714ed8b8df2e7bf23646e5b818981af4d1244914e98a30f9a781"} Nov 03 19:56:25.086357 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.086331 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:25.222040 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.222003 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-openshift-ca\") pod \"abfa53fa-0668-4bb0-9b23-25a41519d799\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " Nov 03 19:56:25.222040 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.222053 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfa53fa-0668-4bb0-9b23-25a41519d799-kube-api-access-s9rkn\" (UniqueName: \"kubernetes.io/projected/abfa53fa-0668-4bb0-9b23-25a41519d799-kube-api-access-s9rkn\") pod \"abfa53fa-0668-4bb0-9b23-25a41519d799\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " Nov 03 19:56:25.222299 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.222088 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-xks-ca\") pod \"abfa53fa-0668-4bb0-9b23-25a41519d799\" (UID: \"abfa53fa-0668-4bb0-9b23-25a41519d799\") " Nov 03 19:56:25.222475 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.222448 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-openshift-ca" pod "abfa53fa-0668-4bb0-9b23-25a41519d799" (UID: "abfa53fa-0668-4bb0-9b23-25a41519d799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:56:25.222946 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.222924 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:25.224207 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.224183 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-xks-ca" pod "abfa53fa-0668-4bb0-9b23-25a41519d799" (UID: "abfa53fa-0668-4bb0-9b23-25a41519d799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:56:25.224297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.224278 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfa53fa-0668-4bb0-9b23-25a41519d799-kube-api-access-s9rkn" pod "abfa53fa-0668-4bb0-9b23-25a41519d799" (UID: "abfa53fa-0668-4bb0-9b23-25a41519d799"). InnerVolumeSpecName "kube-api-access-s9rkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:25.324462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.324378 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rkn\" (UniqueName: \"kubernetes.io/projected/abfa53fa-0668-4bb0-9b23-25a41519d799-kube-api-access-s9rkn\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:25.324462 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.324410 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfa53fa-0668-4bb0-9b23-25a41519d799-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:25.792603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.791892 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" event={"ID":"abfa53fa-0668-4bb0-9b23-25a41519d799","Type":"ContainerDied","Data":"e6151d230bdd22842ffa5147bef4d9321234462e170fad4b2f4f0fb49aa3d5ff"} Nov 03 19:56:25.792603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.791958 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6151d230bdd22842ffa5147bef4d9321234462e170fad4b2f4f0fb49aa3d5ff" Nov 03 19:56:25.792603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:25.792033 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr" Nov 03 19:56:27.850607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:27.846728 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:28.609772 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:28.609718 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:28.665722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:28.665675 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:28.803607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:28.782649 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:28.803607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:28.803087 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:29.689606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.667692 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:29.692871 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.690351 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:29.694154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.694117 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:56:29.726688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.719442 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:56:29.733674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.727219 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:56:29.820573 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.785900 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:29.835487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.786112 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:56:29.841752 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.836403 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:56:29.899145 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:29.846257 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:56:30.181194 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.181152 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:56:30.231666 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.231634 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:56:30.708614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.708473 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 19:56:30.709106 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.708711 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:30.747730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.747678 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:30.840186 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.840147 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:30.847614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.841476 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:30.866012 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.843397 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:56:30.866012 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.844894 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:56:30.866012 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:30.855682 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:56:31.496335 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:31.496285 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:56:31.549620 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:31.548968 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:56:31.549986 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:31.549941 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:56:31.580085 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:31.575123 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:56:31.708796 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:31.708686 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:56:33.049912 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:33.049881 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:56:33.051183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:33.051156 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 19:56:33.857230 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:33.857194 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq"] Nov 03 19:56:33.857718 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:33.857683 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" containerID="cri-o://119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5" gracePeriod=30 Nov 03 19:56:34.130892 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:34.130807 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf"] Nov 03 19:56:34.131359 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:34.131062 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" containerID="cri-o://7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f" gracePeriod=30 Nov 03 19:56:34.140135 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:34.140109 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k"] Nov 03 19:56:34.140504 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:34.140476 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" containerID="cri-o://50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9" gracePeriod=30 Nov 03 19:56:34.409856 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:34.409752 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22"] Nov 03 19:56:34.410030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:34.410000 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" containerID="cri-o://4b89e380c76b7883561d33f35de7f494afdfd5eeb401f6d36ac6a0e3efa4b8b1" gracePeriod=30 Nov 03 19:56:39.375633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.372784 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:56:39.447011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.446639 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-envoy\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.447197 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.447174 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-credential-socket\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.447293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.447235 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istiod-ca-cert\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.447293 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.447282 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-podinfo\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.448254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.447517 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-token\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-token\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.448254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.447553 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-socket\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.448254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.447615 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-kube-api-access-spsxn\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-kube-api-access-spsxn\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.448254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.448039 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-credential-socket" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.448743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.448699 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-socket" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.448743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.448707 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istiod-ca-cert" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:56:39.448937 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.448917 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-data\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.449194 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.449167 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-data" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.449278 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.449235 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-certs\") pod \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\" (UID: \"3bb3b604-88e1-44b9-a4aa-9184c46708fa\") " Nov 03 19:56:39.449448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.449430 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-certs" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.451348 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.451322 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.451454 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.451353 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.451454 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.451372 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.451454 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.451389 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.451454 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.451405 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.462764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.462713 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-podinfo" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:56:39.462764 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.462715 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-envoy" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.462917 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.462771 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-token" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:39.465518 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.465476 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-kube-api-access-spsxn" pod "3bb3b604-88e1-44b9-a4aa-9184c46708fa" (UID: "3bb3b604-88e1-44b9-a4aa-9184c46708fa"). InnerVolumeSpecName "kube-api-access-spsxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:39.553323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.553287 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.553323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.553326 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spsxn\" (UniqueName: \"kubernetes.io/projected/3bb3b604-88e1-44b9-a4aa-9184c46708fa-kube-api-access-spsxn\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.553638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.553343 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.553638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.553357 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb3b604-88e1-44b9-a4aa-9184c46708fa-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.553780 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.553720 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:56:39.591611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.587389 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:56:39.654554 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.654523 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-socket\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.654766 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.654753 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-certs\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.654838 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.654829 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-certs\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.655018 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655005 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-podinfo\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.655119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655104 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-token\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-token\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.655720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655220 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-token\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-token\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.655720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655268 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-podinfo\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.655720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655338 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-kube-api-access-425h9\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-kube-api-access-425h9\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.655720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655436 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-certs" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.655720 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655547 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-credential-socket\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.656097 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.655884 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-data\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.656202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656091 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-socket\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.656202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656150 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-credential-socket\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.656323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656221 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-data\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.656323 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656283 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-envoy\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.656429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656321 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istiod-ca-cert\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.656429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656347 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-credential-socket" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.656429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656385 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istiod-ca-cert\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.656613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656485 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-envoy\") pod \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\" (UID: \"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9\") " Nov 03 19:56:39.656613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656525 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-kube-api-access-zqf7p\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-kube-api-access-zqf7p\") pod \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\" (UID: \"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46\") " Nov 03 19:56:39.656732 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656696 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-socket" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.656988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.656967 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-data" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.657633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.657565 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.657633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.657611 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.657633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.657626 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.657848 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.657640 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.657904 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.657876 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-data" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.658070 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.658052 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-socket" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.658225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.658207 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-credential-socket" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.658498 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.658479 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istiod-ca-cert" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:56:39.659031 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.659006 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-certs" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.659310 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.659288 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istiod-ca-cert" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:56:39.660560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.660510 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-podinfo" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:56:39.662276 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.662243 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-envoy" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.662818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.662788 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-envoy" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:39.663066 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.663033 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-kube-api-access-zqf7p" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "kube-api-access-zqf7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:39.668913 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.668878 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-podinfo" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:56:39.669008 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.668950 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-kube-api-access-425h9" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "kube-api-access-425h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:39.669122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.669103 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-token" pod "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" (UID: "447a4a3a-9e30-44ae-b196-f9c2a7b77fc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:39.672164 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.672134 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-token" pod "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" (UID: "6eb5bc4d-59ea-4f08-ba3c-459eea50fe46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:39.759952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.759913 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqf7p\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-kube-api-access-zqf7p\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760245 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760224 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760370 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760392 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760407 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760420 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760435 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425h9\" (UniqueName: \"kubernetes.io/projected/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-kube-api-access-425h9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760449 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760461 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760474 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760488 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760501 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760514 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.760542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.760527 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:39.761389 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.761363 4314 generic.go:356] "Generic (PLEG): container finished" podID="62d72718-b839-4e03-a321-fdb818d157e7" containerID="4b89e380c76b7883561d33f35de7f494afdfd5eeb401f6d36ac6a0e3efa4b8b1" exitCode=0 Nov 03 19:56:39.761479 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.761461 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" event={"ID":"62d72718-b839-4e03-a321-fdb818d157e7","Type":"ContainerDied","Data":"4b89e380c76b7883561d33f35de7f494afdfd5eeb401f6d36ac6a0e3efa4b8b1"} Nov 03 19:56:39.763396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.763369 4314 generic.go:356] "Generic (PLEG): container finished" podID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerID="119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5" exitCode=0 Nov 03 19:56:39.763520 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.763487 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" event={"ID":"3bb3b604-88e1-44b9-a4aa-9184c46708fa","Type":"ContainerDied","Data":"119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5"} Nov 03 19:56:39.763643 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.763552 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" event={"ID":"3bb3b604-88e1-44b9-a4aa-9184c46708fa","Type":"ContainerDied","Data":"241b1e2613ed896c70beb4795d4860a34ad253562e2b09927bf201e4f6a02cc2"} Nov 03 19:56:39.763707 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.763643 4314 scope.go:122] "RemoveContainer" containerID="119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5" Nov 03 19:56:39.763993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.763976 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq" Nov 03 19:56:39.769112 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.769087 4314 generic.go:356] "Generic (PLEG): container finished" podID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerID="7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f" exitCode=0 Nov 03 19:56:39.769341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.769322 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" event={"ID":"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46","Type":"ContainerDied","Data":"7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f"} Nov 03 19:56:39.769476 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.769462 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" event={"ID":"6eb5bc4d-59ea-4f08-ba3c-459eea50fe46","Type":"ContainerDied","Data":"b79261f3051e02560061591029ce38f6e96322bdd048f8195d4b6ac5301eef32"} Nov 03 19:56:39.769733 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.769717 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf" Nov 03 19:56:39.783432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.782703 4314 generic.go:356] "Generic (PLEG): container finished" podID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerID="50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9" exitCode=0 Nov 03 19:56:39.783432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.782787 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" event={"ID":"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9","Type":"ContainerDied","Data":"50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9"} Nov 03 19:56:39.783432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.782827 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" event={"ID":"447a4a3a-9e30-44ae-b196-f9c2a7b77fc9","Type":"ContainerDied","Data":"765b34cac7cb06440202618dc0c29487f829dcfd88bc5ee2f1fc616a07843921"} Nov 03 19:56:39.783432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.782900 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k" Nov 03 19:56:39.826731 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.826704 4314 scope.go:122] "RemoveContainer" containerID="119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5" Nov 03 19:56:39.828155 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:56:39.827719 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5\": container with ID starting with 119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5 not found: ID does not exist" containerID="119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5" Nov 03 19:56:39.828155 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.827758 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5"} err="failed to get container status \"119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5\": rpc error: code = NotFound desc = could not find container \"119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5\": container with ID starting with 119efaded3c953d9ebcfd7e5bfda9870cabbda72d511e771006b93258d3bf0b5 not found: ID does not exist" Nov 03 19:56:39.828155 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.827785 4314 scope.go:122] "RemoveContainer" containerID="7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f" Nov 03 19:56:39.839688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.839665 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq"] Nov 03 19:56:39.846305 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.846282 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-qfvzq"] Nov 03 19:56:39.851113 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.851095 4314 scope.go:122] "RemoveContainer" containerID="7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f" Nov 03 19:56:39.851271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.851250 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k"] Nov 03 19:56:39.851463 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:56:39.851433 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f\": container with ID starting with 7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f not found: ID does not exist" containerID="7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f" Nov 03 19:56:39.851566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.851508 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f"} err="failed to get container status \"7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f\": rpc error: code = NotFound desc = could not find container \"7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f\": container with ID starting with 7734af899530673e75baa2237aa18c59eb7a6f6eaa9813bec5e3a6b1c3b2bc5f not found: ID does not exist" Nov 03 19:56:39.851566 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.851536 4314 scope.go:122] "RemoveContainer" containerID="50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9" Nov 03 19:56:39.857769 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.857721 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fgh2k"] Nov 03 19:56:39.876560 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.876505 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf"] Nov 03 19:56:39.906617 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:39.881564 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-2fwrf"] Nov 03 19:56:40.003684 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.002741 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:56:40.014472 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.013915 4314 scope.go:122] "RemoveContainer" containerID="bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51" Nov 03 19:56:40.039217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.039062 4314 scope.go:122] "RemoveContainer" containerID="50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9" Nov 03 19:56:40.039593 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:56:40.039426 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9\": container with ID starting with 50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9 not found: ID does not exist" containerID="50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9" Nov 03 19:56:40.039593 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.039467 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9"} err="failed to get container status \"50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9\": rpc error: code = NotFound desc = could not find container \"50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9\": container with ID starting with 50d0800df4599a54511ad8d283bcc0837def3cd7ad578c552677b329af39a0d9 not found: ID does not exist" Nov 03 19:56:40.039593 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.039495 4314 scope.go:122] "RemoveContainer" containerID="bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51" Nov 03 19:56:40.039932 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:56:40.039874 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51\": container with ID starting with bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51 not found: ID does not exist" containerID="bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51" Nov 03 19:56:40.039932 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.039907 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51"} err="failed to get container status \"bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51\": rpc error: code = NotFound desc = could not find container \"bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51\": container with ID starting with bb7d6c925f72be623a47a47b3a32e654e0a7daf58968e43e0174143576ea3a51 not found: ID does not exist" Nov 03 19:56:40.085721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.085683 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-kube-api-access-qkt48\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-kube-api-access-qkt48\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.085925 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.085741 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-credential-socket\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.085925 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.085814 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-istio-token\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-istio-token\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.086068 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086033 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-certs\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.086124 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086072 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-socket\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.086195 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086179 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d72718-b839-4e03-a321-fdb818d157e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d72718-b839-4e03-a321-fdb818d157e7-istiod-ca-cert\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.086304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086289 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-data\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.086362 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086345 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-envoy\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.086418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086352 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-certs" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:40.086418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086385 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-socket" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:40.086418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086403 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62d72718-b839-4e03-a321-fdb818d157e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d72718-b839-4e03-a321-fdb818d157e7-istio-podinfo\") pod \"62d72718-b839-4e03-a321-fdb818d157e7\" (UID: \"62d72718-b839-4e03-a321-fdb818d157e7\") " Nov 03 19:56:40.086418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086403 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-credential-socket" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:40.086642 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086602 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-data" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:40.086717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.086695 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d72718-b839-4e03-a321-fdb818d157e7-istiod-ca-cert" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:56:40.087440 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.087320 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.087440 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.087346 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.087440 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.087360 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d72718-b839-4e03-a321-fdb818d157e7-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.087440 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.087372 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.087440 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.087384 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.092341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.089821 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-kube-api-access-qkt48" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "kube-api-access-qkt48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:40.092341 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.090082 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-istio-token" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:56:40.093285 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.093237 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62d72718-b839-4e03-a321-fdb818d157e7-istio-podinfo" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:56:40.093382 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.093331 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-envoy" pod "62d72718-b839-4e03-a321-fdb818d157e7" (UID: "62d72718-b839-4e03-a321-fdb818d157e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:56:40.189805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.189713 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d72718-b839-4e03-a321-fdb818d157e7-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.189805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.189744 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d72718-b839-4e03-a321-fdb818d157e7-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.189805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.189766 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkt48\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-kube-api-access-qkt48\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.189805 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.189783 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d72718-b839-4e03-a321-fdb818d157e7-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:56:40.320970 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.320897 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx"] Nov 03 19:56:40.321874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321855 4314 cpu_manager.go:414] "containerMap: removing container" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321875 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321909 4314 cpu_manager.go:414] "containerMap: removing container" podUID="abfa53fa-0668-4bb0-9b23-25a41519d799" containerName="revoke-keys" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321915 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfa53fa-0668-4bb0-9b23-25a41519d799" containerName="revoke-keys" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321927 4314 cpu_manager.go:414] "containerMap: removing container" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321933 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321942 4314 cpu_manager.go:414] "containerMap: removing container" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321946 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321956 4314 cpu_manager.go:414] "containerMap: removing container" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" Nov 03 19:56:40.321966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321960 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321977 4314 cpu_manager.go:414] "containerMap: removing container" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.321981 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.322088 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.322100 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" containerName="istio-proxy" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.322111 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfa53fa-0668-4bb0-9b23-25a41519d799" containerName="revoke-keys" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.322118 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" containerName="istio-proxy" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.322124 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" containerName="istio-proxy" Nov 03 19:56:40.322233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.322132 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d72718-b839-4e03-a321-fdb818d157e7" containerName="istio-proxy" Nov 03 19:56:40.325972 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.325950 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.331271 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.331247 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx"] Nov 03 19:56:40.332686 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.332662 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d12868-dockercfg-qswff\"" Nov 03 19:56:40.332807 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.332703 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d12868\"" Nov 03 19:56:40.375401 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.375275 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm"] Nov 03 19:56:40.402603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.399660 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92ce758-b31d-46a2-990f-4f222c049a1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.402603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.399850 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5j4t\" (UniqueName: \"kubernetes.io/projected/b92ce758-b31d-46a2-990f-4f222c049a1f-kube-api-access-r5j4t\") pod \"payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.411206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.411179 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm"] Nov 03 19:56:40.411542 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.411518 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:40.535215 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.529976 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92ce758-b31d-46a2-990f-4f222c049a1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.583671 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.564858 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5j4t\" (UniqueName: \"kubernetes.io/projected/b92ce758-b31d-46a2-990f-4f222c049a1f-kube-api-access-r5j4t\") pod \"payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.583671 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.566243 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92ce758-b31d-46a2-990f-4f222c049a1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.837664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.832983 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqcv\" (UniqueName: \"kubernetes.io/projected/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-kube-api-access-knqcv\") pod \"payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:40.837664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.833209 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:40.872236 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.872204 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5j4t\" (UniqueName: \"kubernetes.io/projected/b92ce758-b31d-46a2-990f-4f222c049a1f-kube-api-access-r5j4t\") pod \"payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.877253 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.877226 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:40.893900 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.893873 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" Nov 03 19:56:40.894569 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.894543 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb3b604-88e1-44b9-a4aa-9184c46708fa" path="/var/lib/kubelet/pods/3bb3b604-88e1-44b9-a4aa-9184c46708fa/volumes" Nov 03 19:56:40.895375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.895358 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447a4a3a-9e30-44ae-b196-f9c2a7b77fc9" path="/var/lib/kubelet/pods/447a4a3a-9e30-44ae-b196-f9c2a7b77fc9/volumes" Nov 03 19:56:40.896287 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.896249 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb5bc4d-59ea-4f08-ba3c-459eea50fe46" path="/var/lib/kubelet/pods/6eb5bc4d-59ea-4f08-ba3c-459eea50fe46/volumes" Nov 03 19:56:40.897662 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.897642 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22" event={"ID":"62d72718-b839-4e03-a321-fdb818d157e7","Type":"ContainerDied","Data":"6a23b200d59f9821dc001c5812795d75ddf4bce7a1a6fafed77e1e22503cb38e"} Nov 03 19:56:40.897806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.897794 4314 scope.go:122] "RemoveContainer" containerID="4b89e380c76b7883561d33f35de7f494afdfd5eeb401f6d36ac6a0e3efa4b8b1" Nov 03 19:56:40.902969 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.902947 4314 generic.go:356] "Generic (PLEG): container finished" podID="fa2d0438-5595-42d4-821e-5618f05382fd" containerID="14de153be85fc5e6443a99cc5621f5e19a9867e9f647f414a2b55368494e3463" exitCode=0 Nov 03 19:56:40.903233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.903193 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" event={"ID":"fa2d0438-5595-42d4-821e-5618f05382fd","Type":"ContainerDied","Data":"14de153be85fc5e6443a99cc5621f5e19a9867e9f647f414a2b55368494e3463"} Nov 03 19:56:40.903853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.903833 4314 scope.go:122] "RemoveContainer" containerID="14de153be85fc5e6443a99cc5621f5e19a9867e9f647f414a2b55368494e3463" Nov 03 19:56:40.906453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.906414 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:56:40.906632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.906511 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:56:40.906632 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.906559 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:56:40.991250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.980910 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:40.991250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.981304 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqcv\" (UniqueName: \"kubernetes.io/projected/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-kube-api-access-knqcv\") pod \"payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:40.991250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.982958 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:40.991250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.983330 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22"] Nov 03 19:56:40.991250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.985251 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-kfw22"] Nov 03 19:56:40.991250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:40.990063 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqcv\" (UniqueName: \"kubernetes.io/projected/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-kube-api-access-knqcv\") pod \"payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:41.211905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.206672 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx"] Nov 03 19:56:41.214133 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.213154 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:41.536460 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:56:41.536429 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb52a4a_1ec1_4a25_9ee6_c29d69884d5b.slice/crio-6630cd1a02ae6a78ddc47d5cc17ada91d547367ad075c5b45a0c7524d7a7120a WatchSource:0}: Error finding container 6630cd1a02ae6a78ddc47d5cc17ada91d547367ad075c5b45a0c7524d7a7120a: Status 404 returned error can't find the container with id 6630cd1a02ae6a78ddc47d5cc17ada91d547367ad075c5b45a0c7524d7a7120a Nov 03 19:56:41.537071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.537049 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm"] Nov 03 19:56:41.831254 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.831114 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:56:41.913483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.913448 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" event={"ID":"fa2d0438-5595-42d4-821e-5618f05382fd","Type":"ContainerStarted","Data":"b70707b1082ef4b21aff627053f4896b50eba458f547b09888b0222db90a69ff"} Nov 03 19:56:41.915349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.915321 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" event={"ID":"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b","Type":"ContainerStarted","Data":"732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7"} Nov 03 19:56:41.915464 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.915367 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" event={"ID":"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b","Type":"ContainerStarted","Data":"6630cd1a02ae6a78ddc47d5cc17ada91d547367ad075c5b45a0c7524d7a7120a"} Nov 03 19:56:41.915464 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.915401 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:41.918085 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.918062 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" event={"ID":"b92ce758-b31d-46a2-990f-4f222c049a1f","Type":"ContainerStarted","Data":"c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0"} Nov 03 19:56:41.918182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.918111 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" event={"ID":"b92ce758-b31d-46a2-990f-4f222c049a1f","Type":"ContainerStarted","Data":"5f0dc03cb55f7cb168d6cfd5bf094c2ac6a4f51251101169050e78273ee02e75"} Nov 03 19:56:41.918182 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.918142 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:41.987673 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:41.987609 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" podStartSLOduration=1.987574953 podStartE2EDuration="1.987574953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:41.981053355 +0000 UTC m=+1509.685892696" watchObservedRunningTime="2026-11-03 19:56:41.987574953 +0000 UTC m=+1509.692414294" Nov 03 19:56:42.013601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:42.010568 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" podStartSLOduration=2.010552815 podStartE2EDuration="2.010552815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:42.010042599 +0000 UTC m=+1509.714881945" watchObservedRunningTime="2026-11-03 19:56:42.010552815 +0000 UTC m=+1509.715392154" Nov 03 19:56:42.254096 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:42.254060 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:56:42.255201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:42.255173 4314 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:56:42.255317 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:42.255232 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 19:56:42.432055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:42.432023 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d72718-b839-4e03-a321-fdb818d157e7" path="/var/lib/kubelet/pods/62d72718-b839-4e03-a321-fdb818d157e7/volumes" Nov 03 19:56:43.254011 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:43.253968 4314 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:56:43.254425 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:43.254025 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 19:56:44.254217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:44.254179 4314 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:56:44.254872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:44.254244 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 19:56:45.253887 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:45.253854 4314 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:56:45.254075 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:45.253917 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 19:56:46.253399 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:46.253364 4314 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:56:46.253919 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:46.253433 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 19:56:47.253692 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:47.253553 4314 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 19:56:47.254122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:47.253724 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 19:56:48.259745 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:48.259714 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:56:48.260264 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:48.260240 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:56:48.260988 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:48.260971 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 19:56:51.288639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.256120 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl"] Nov 03 19:56:51.298691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.298595 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl"] Nov 03 19:56:51.299119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.298959 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.309960 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.309934 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c18bea-openshift-default-dockercfg-bmhwm\"" Nov 03 19:56:51.332079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.332044 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-certs\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.332284 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.332265 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-credential-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.332482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.332464 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-envoy\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.332730 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.332712 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-token\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.332953 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.332935 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-podinfo\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.333071 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.333057 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-data\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.333227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.333212 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd6sm\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-kube-api-access-pd6sm\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.333395 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.333380 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.333637 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.333620 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istiod-ca-cert\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.439324 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.439291 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-certs\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.439721 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.439573 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-credential-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.439944 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.439921 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-envoy\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.440144 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.440123 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-token\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.440453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.440434 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-podinfo\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.440636 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.440618 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-data\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.440808 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.440780 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd6sm\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-kube-api-access-pd6sm\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.440966 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.440948 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.441190 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.441173 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istiod-ca-cert\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.442196 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.442173 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istiod-ca-cert\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.478540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.478497 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.483843 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.478825 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-certs\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.483843 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.479071 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-credential-socket\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.508872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.498975 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-podinfo\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.508872 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.499566 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-data\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.539009 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.538748 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-token\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.549952 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.549911 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-envoy\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.551411 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.551376 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd6sm\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-kube-api-access-pd6sm\") pod \"e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.630757 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:51.630721 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:51.991754 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:56:51.991717 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:56:52.026311 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:56:52.026283 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb09d7b_edcd_4f5b_a297_b700f4b32b59.slice/crio-52f6fc3b0d265a4d21ce391011cd6a45ccd4e312114dffd854789cdca63e25df WatchSource:0}: Error finding container 52f6fc3b0d265a4d21ce391011cd6a45ccd4e312114dffd854789cdca63e25df: Status 404 returned error can't find the container with id 52f6fc3b0d265a4d21ce391011cd6a45ccd4e312114dffd854789cdca63e25df Nov 03 19:56:52.026565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.026541 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl"] Nov 03 19:56:52.029565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.029531 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:56:52.029676 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.029644 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:56:52.029714 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.029684 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:56:52.274174 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.274129 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" event={"ID":"7eb09d7b-edcd-4f5b-a297-b700f4b32b59","Type":"ContainerStarted","Data":"c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d"} Nov 03 19:56:52.274375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.274200 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" event={"ID":"7eb09d7b-edcd-4f5b-a297-b700f4b32b59","Type":"ContainerStarted","Data":"52f6fc3b0d265a4d21ce391011cd6a45ccd4e312114dffd854789cdca63e25df"} Nov 03 19:56:52.303525 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.303408 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podStartSLOduration=1.303390608 podStartE2EDuration="1.303390608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:52.296009953 +0000 UTC m=+1520.000849296" watchObservedRunningTime="2026-11-03 19:56:52.303390608 +0000 UTC m=+1520.008229949" Nov 03 19:56:52.631192 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.631117 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:52.930097 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.930017 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 19:56:52.930097 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:52.930084 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 19:56:53.637534 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:53.637497 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:54.293389 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:54.293354 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:54.294244 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:54.294220 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:56:55.972251 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:55.972218 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn"] Nov 03 19:56:55.977408 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:55.977387 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:55.990897 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:55.987161 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn"] Nov 03 19:56:56.003004 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.001488 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs"] Nov 03 19:56:56.008358 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.008333 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.026602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.017642 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c18bea-dockercfg-72c6b\"" Nov 03 19:56:56.034213 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.033749 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs"] Nov 03 19:56:56.059312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.059278 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7z4w\" (UniqueName: \"kubernetes.io/projected/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-kube-api-access-b7z4w\") pod \"payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:56.059465 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.059355 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vtmr\" (UniqueName: \"kubernetes.io/projected/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-kube-api-access-7vtmr\") pod \"payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.059530 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.059505 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:56.059711 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.059689 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.077943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.077918 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c18bea\"" Nov 03 19:56:56.160914 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.160879 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.161130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.161112 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7z4w\" (UniqueName: \"kubernetes.io/projected/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-kube-api-access-b7z4w\") pod \"payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:56.161206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.161164 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vtmr\" (UniqueName: \"kubernetes.io/projected/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-kube-api-access-7vtmr\") pod \"payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.161265 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.161245 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:56.161693 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.161672 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.162233 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.162208 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:56.184459 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.184307 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7z4w\" (UniqueName: \"kubernetes.io/projected/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-kube-api-access-b7z4w\") pod \"payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:56.185281 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.185038 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vtmr\" (UniqueName: \"kubernetes.io/projected/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-kube-api-access-7vtmr\") pod \"payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.321074 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.319892 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:56.337136 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.337099 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:56.956757 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:56:56.956722 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70b7ba4_f07c_4b53_89b9_80bb29186ad3.slice/crio-4f2cdea11fdfe2a0c87265481b03a7c57b20f18d72335863a1211254b99cf570 WatchSource:0}: Error finding container 4f2cdea11fdfe2a0c87265481b03a7c57b20f18d72335863a1211254b99cf570: Status 404 returned error can't find the container with id 4f2cdea11fdfe2a0c87265481b03a7c57b20f18d72335863a1211254b99cf570 Nov 03 19:56:56.956861 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.956762 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn"] Nov 03 19:56:56.979958 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:56.979927 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs"] Nov 03 19:56:56.985478 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:56:56.985455 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc63876_7c5c_4a54_8227_d78f36e4e4d9.slice/crio-b9e8fa2b1173839ce0c143851f4f0c28af33acaadc89b8d1c3e21d5dace08c11 WatchSource:0}: Error finding container b9e8fa2b1173839ce0c143851f4f0c28af33acaadc89b8d1c3e21d5dace08c11: Status 404 returned error can't find the container with id b9e8fa2b1173839ce0c143851f4f0c28af33acaadc89b8d1c3e21d5dace08c11 Nov 03 19:56:57.455174 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.455131 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" event={"ID":"5cc63876-7c5c-4a54-8227-d78f36e4e4d9","Type":"ContainerStarted","Data":"e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657"} Nov 03 19:56:57.455358 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.455196 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:56:57.455358 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.455220 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" event={"ID":"5cc63876-7c5c-4a54-8227-d78f36e4e4d9","Type":"ContainerStarted","Data":"b9e8fa2b1173839ce0c143851f4f0c28af33acaadc89b8d1c3e21d5dace08c11"} Nov 03 19:56:57.456998 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.456969 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" event={"ID":"e70b7ba4-f07c-4b53-89b9-80bb29186ad3","Type":"ContainerStarted","Data":"9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711"} Nov 03 19:56:57.457105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.457012 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" event={"ID":"e70b7ba4-f07c-4b53-89b9-80bb29186ad3","Type":"ContainerStarted","Data":"4f2cdea11fdfe2a0c87265481b03a7c57b20f18d72335863a1211254b99cf570"} Nov 03 19:56:57.457105 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.457067 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:56:57.492390 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.492333 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" podStartSLOduration=2.49231966 podStartE2EDuration="2.49231966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:57.490678922 +0000 UTC m=+1525.195518262" watchObservedRunningTime="2026-11-03 19:56:57.49231966 +0000 UTC m=+1525.197158999" Nov 03 19:56:57.524696 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:56:57.524643 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" podStartSLOduration=2.524620494 podStartE2EDuration="2.524620494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:57.519079136 +0000 UTC m=+1525.223918503" watchObservedRunningTime="2026-11-03 19:56:57.524620494 +0000 UTC m=+1525.229459834" Nov 03 19:57:02.034099 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.034055 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7"] Nov 03 19:57:02.039363 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.039337 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.060434 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.060406 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7"] Nov 03 19:57:02.065814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.065788 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d12868-serving-cert\"" Nov 03 19:57:02.131000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.130966 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-maas-api-tls\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.131157 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.131129 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0afcbf-c491-43d5-a545-08b5b7174777-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.131205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.131165 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-metrics-certs\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.131392 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.131366 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4b6\" (UniqueName: \"kubernetes.io/projected/0b0afcbf-c491-43d5-a545-08b5b7174777-kube-api-access-wj4b6\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.256615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.255041 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-maas-api-tls\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.256615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.255486 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0afcbf-c491-43d5-a545-08b5b7174777-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.256615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.255566 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-metrics-certs\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.256615 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.255871 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4b6\" (UniqueName: \"kubernetes.io/projected/0b0afcbf-c491-43d5-a545-08b5b7174777-kube-api-access-wj4b6\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.261064 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.261009 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-maas-api-tls\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.262015 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.261975 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0afcbf-c491-43d5-a545-08b5b7174777-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.267158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.265792 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4b6\" (UniqueName: \"kubernetes.io/projected/0b0afcbf-c491-43d5-a545-08b5b7174777-kube-api-access-wj4b6\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.267158 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.267043 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-metrics-certs\") pod \"maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.371711 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.371629 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:02.828801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:02.823236 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7"] Nov 03 19:57:03.543114 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:03.543062 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" event={"ID":"0b0afcbf-c491-43d5-a545-08b5b7174777","Type":"ContainerStarted","Data":"1b486276f47e01001d23bcc6cf2e2214340d7a1c21e73d7a2090f88d8a558820"} Nov 03 19:57:04.553712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:04.553669 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" event={"ID":"0b0afcbf-c491-43d5-a545-08b5b7174777","Type":"ContainerStarted","Data":"9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a"} Nov 03 19:57:04.554185 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:04.553810 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:04.572385 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:04.572335 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" podStartSLOduration=1.988968471 podStartE2EDuration="2.572322867s" totalImagesPullingTime="583.354396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:04.571008318 +0000 UTC m=+1532.275847682" watchObservedRunningTime="2026-11-03 19:57:04.572322867 +0000 UTC m=+1532.277162207" Nov 03 19:57:08.469420 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:08.469387 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 19:57:08.469928 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:08.469457 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 19:57:10.572638 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:10.572604 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 19:57:12.703679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:12.703642 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr"] Nov 03 19:57:12.710210 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:12.710181 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d0987-wzmhr"] Nov 03 19:57:14.431646 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:14.431615 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfa53fa-0668-4bb0-9b23-25a41519d799" path="/var/lib/kubelet/pods/abfa53fa-0668-4bb0-9b23-25a41519d799/volumes" Nov 03 19:57:19.389422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:19.386782 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4"] Nov 03 19:57:19.389422 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:19.387094 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" containerID="cri-o://e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591" gracePeriod=30 Nov 03 19:57:20.497128 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:20.497090 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7"] Nov 03 19:57:20.498300 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:20.497938 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" containerID="cri-o://75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9" gracePeriod=30 Nov 03 19:57:22.276926 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:57:22.276882 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:57:24.748712 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.748685 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:57:24.933208 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.933168 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-credential-socket\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.933700 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.933657 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-token\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-token\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.933960 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.933942 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-certs\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.934888 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.934282 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-envoy\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.939688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.939663 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istiod-ca-cert\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.944750 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.944728 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-socket\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.945372 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.945350 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-kube-api-access-cngr9\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-kube-api-access-cngr9\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.945778 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.945752 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-data\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.945982 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.945964 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-podinfo\") pod \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\" (UID: \"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f\") " Nov 03 19:57:24.963000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.944922 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-credential-socket" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:24.963000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.958559 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-socket" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:24.963000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.959030 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-data" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:24.963000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.959607 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:24.963000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.959629 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:24.963000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.959644 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:24.963000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.962821 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istiod-ca-cert" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.966657 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-certs" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.966711 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x"] Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.968030 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.968048 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:24.968243 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerName="istio-proxy" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.007944 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-kube-api-access-cngr9" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "kube-api-access-cngr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.009871 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-envoy" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.013070 4314 generic.go:356] "Generic (PLEG): container finished" podID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" containerID="e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591" exitCode=0 Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.013325 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.025784 4314 patch_prober.go:29] interesting pod/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:57:25.026183 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.025852 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:57:25.085633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.063967 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-podinfo" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:57:25.123616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.098518 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cngr9\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-kube-api-access-cngr9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:25.123616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.098557 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:25.123616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.098573 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:25.123616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.098605 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:25.123616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.098618 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.145269 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" event={"ID":"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f","Type":"ContainerDied","Data":"e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591"} Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.145334 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4" event={"ID":"a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f","Type":"ContainerDied","Data":"0d4a24c9cbd99999eb5336fb34128334a58351fd3eee8926b79379cb05d70ef7"} Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.145384 4314 scope.go:122] "RemoveContainer" containerID="e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591" Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.145699 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.146953 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-token" pod "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" (UID: "a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.170370 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x"] Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.188968 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2747a4-openshift-default-dockercfg-vlbl2\"" Nov 03 19:57:25.227852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.206965 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:25.229801 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.229773 4314 scope.go:122] "RemoveContainer" containerID="e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591" Nov 03 19:57:25.230422 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:57:25.230393 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591\": container with ID starting with e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591 not found: ID does not exist" containerID="e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591" Nov 03 19:57:25.230548 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.230430 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591"} err="failed to get container status \"e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591\": rpc error: code = NotFound desc = could not find container \"e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591\": container with ID starting with e9f0f28636f93679546104cae48af420c33d98d971402cfc17e2933deadec591 not found: ID does not exist" Nov 03 19:57:25.308568 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.308527 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-istio-token\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.308781 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.308710 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-kube-api-access-m7zv2\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.308870 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.308852 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-envoy\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.308943 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.308898 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-socket\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.309028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.309014 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5374374-e149-4721-beaa-b32f10b11ea9-istiod-ca-cert\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.309124 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.309110 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-certs\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.309202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.309188 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5374374-e149-4721-beaa-b32f10b11ea9-istio-podinfo\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.309429 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.309404 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-credential-socket\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.309511 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.309460 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-data\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.341927 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.341887 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4"] Nov 03 19:57:25.346800 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.346773 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-2k4f4"] Nov 03 19:57:25.418247 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.418206 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-certs\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.419122 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.419098 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5374374-e149-4721-beaa-b32f10b11ea9-istio-podinfo\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.419979 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.419957 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-credential-socket\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.420546 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.420526 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-data\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.420787 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.420765 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-istio-token\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.420984 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.420966 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-kube-api-access-m7zv2\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.421193 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.421175 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-envoy\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.421328 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.421310 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-socket\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.421565 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.421547 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5374374-e149-4721-beaa-b32f10b11ea9-istiod-ca-cert\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.422477 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.422454 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5374374-e149-4721-beaa-b32f10b11ea9-istiod-ca-cert\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.422625 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.419007 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-certs\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.422956 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.422938 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-data\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.423067 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.420469 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-credential-socket\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.424167 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.424143 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-socket\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.436605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.428565 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5374374-e149-4721-beaa-b32f10b11ea9-istio-podinfo\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.444611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.442653 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-envoy\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.454601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.452952 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-istio-token\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.454601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.453889 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-kube-api-access-m7zv2\") pod \"e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.506365 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.504982 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:25.815730 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:57:25.815698 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5374374_e149_4721_beaa_b32f10b11ea9.slice/crio-29b1f02c548d74874c03007460c8cc027ac6e069a64dc5dec382a5ad084428dc WatchSource:0}: Error finding container 29b1f02c548d74874c03007460c8cc027ac6e069a64dc5dec382a5ad084428dc: Status 404 returned error can't find the container with id 29b1f02c548d74874c03007460c8cc027ac6e069a64dc5dec382a5ad084428dc Nov 03 19:57:25.816686 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.816571 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x"] Nov 03 19:57:25.820000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.819958 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:25.820081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.820063 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:25.820123 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.820107 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:25.843869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.843845 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:57:25.934223 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934191 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-envoy\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934376 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934283 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-podinfo\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934495 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934473 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-certs\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934531 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-kube-api-access-5cd78\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-kube-api-access-5cd78\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934617 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-data\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934737 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934707 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-socket\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934747 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-token\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-token\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934835 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istiod-ca-cert\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.934921 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.934905 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-credential-socket\") pod \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\" (UID: \"a422e62b-e92c-4dff-8ef6-5722d98b7e06\") " Nov 03 19:57:25.935573 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.935542 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-socket" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:25.935912 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.935886 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-credential-socket" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:25.936002 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.935913 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:25.936309 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.936126 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-data" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:25.936812 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.936568 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istiod-ca-cert" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:57:25.936992 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.936958 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-certs" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:25.937798 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.937658 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-envoy" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:57:25.939354 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.939328 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-token" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:25.939456 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.939434 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-podinfo" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:57:25.939523 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:25.939506 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-kube-api-access-5cd78" pod "a422e62b-e92c-4dff-8ef6-5722d98b7e06" (UID: "a422e62b-e92c-4dff-8ef6-5722d98b7e06"). InnerVolumeSpecName "kube-api-access-5cd78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:57:26.028276 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.028238 4314 generic.go:356] "Generic (PLEG): container finished" podID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerID="75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9" exitCode=0 Nov 03 19:57:26.028443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.028338 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" Nov 03 19:57:26.028443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.028360 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" event={"ID":"a422e62b-e92c-4dff-8ef6-5722d98b7e06","Type":"ContainerDied","Data":"75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9"} Nov 03 19:57:26.028443 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.028431 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7" event={"ID":"a422e62b-e92c-4dff-8ef6-5722d98b7e06","Type":"ContainerDied","Data":"53153730c72c6922c3b4f223f1440932248b8fda651a0c975df90a1653fa6c71"} Nov 03 19:57:26.028601 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.028457 4314 scope.go:122] "RemoveContainer" containerID="75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9" Nov 03 19:57:26.031783 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.031756 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" event={"ID":"f5374374-e149-4721-beaa-b32f10b11ea9","Type":"ContainerStarted","Data":"849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41"} Nov 03 19:57:26.031918 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.031802 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" event={"ID":"f5374374-e149-4721-beaa-b32f10b11ea9","Type":"ContainerStarted","Data":"29b1f02c548d74874c03007460c8cc027ac6e069a64dc5dec382a5ad084428dc"} Nov 03 19:57:26.037371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037342 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.037371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037365 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.037371 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037375 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.037618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037384 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.037618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037394 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.037618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037403 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cd78\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-kube-api-access-5cd78\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.037618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037413 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.037618 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.037421 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a422e62b-e92c-4dff-8ef6-5722d98b7e06-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:57:26.042454 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.042284 4314 scope.go:122] "RemoveContainer" containerID="89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f" Nov 03 19:57:26.054243 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.054221 4314 scope.go:122] "RemoveContainer" containerID="75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9" Nov 03 19:57:26.054494 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:57:26.054471 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9\": container with ID starting with 75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9 not found: ID does not exist" containerID="75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9" Nov 03 19:57:26.054545 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.054505 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9"} err="failed to get container status \"75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9\": rpc error: code = NotFound desc = could not find container \"75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9\": container with ID starting with 75c6b7a6cf2930bc593c86d9a1ee9bab9da53ad354a29cc10cff56ff440f8ea9 not found: ID does not exist" Nov 03 19:57:26.054545 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.054528 4314 scope.go:122] "RemoveContainer" containerID="89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f" Nov 03 19:57:26.054849 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:57:26.054830 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f\": container with ID starting with 89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f not found: ID does not exist" containerID="89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f" Nov 03 19:57:26.054890 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.054855 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f"} err="failed to get container status \"89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f\": rpc error: code = NotFound desc = could not find container \"89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f\": container with ID starting with 89b08bf43eac63cfc2bdf0dec09a669a0afcbd41922a794109e538681074969f not found: ID does not exist" Nov 03 19:57:26.056459 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.056382 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podStartSLOduration=5.056363623 podStartE2EDuration="5.056363623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:26.050201878 +0000 UTC m=+1553.755041219" watchObservedRunningTime="2026-11-03 19:57:26.056363623 +0000 UTC m=+1553.761202964" Nov 03 19:57:26.082688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.081833 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7"] Nov 03 19:57:26.089378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.089345 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0d0987-openshift-default-7bb6d56c95-4d2c7"] Nov 03 19:57:26.431540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.431466 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f" path="/var/lib/kubelet/pods/a39f0f54-a9ca-4bdd-81ef-2c47e4d00c0f/volumes" Nov 03 19:57:26.432145 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.432119 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" path="/var/lib/kubelet/pods/a422e62b-e92c-4dff-8ef6-5722d98b7e06/volumes" Nov 03 19:57:26.506055 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.506021 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:26.512549 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:26.512525 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:27.043510 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:27.043478 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:27.044386 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:27.044363 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:57:28.670679 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:28.670638 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:57:31.423605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:31.407675 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:31.588640 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:31.582776 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:31.604603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:31.598665 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:33.057602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.057243 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt"] Nov 03 19:57:33.085603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.084773 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" Nov 03 19:57:33.085603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.084824 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" Nov 03 19:57:33.085603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.084872 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" Nov 03 19:57:33.085603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.084881 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" Nov 03 19:57:33.085603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.085526 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" Nov 03 19:57:33.085603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.085555 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a422e62b-e92c-4dff-8ef6-5722d98b7e06" containerName="istio-proxy" Nov 03 19:57:33.153688 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.120117 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.415703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.398726 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:33.522361 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.500254 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7a62d4\"/\"openshift-service-ca.crt\"" Nov 03 19:57:33.522361 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.500377 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7a62d4\"/\"kube-root-ca.crt\"" Nov 03 19:57:33.522361 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.500550 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7a62d4\"/\"default-dockercfg-q2cfx\"" Nov 03 19:57:33.636606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.635808 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468d654-86d4-4e80-a134-3c566229a9f5-tls-certs\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.636606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.635891 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-tmp-dir\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.636606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.635943 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-dshm\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.636606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.636220 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9m9f\" (UniqueName: \"kubernetes.io/projected/f468d654-86d4-4e80-a134-3c566229a9f5-kube-api-access-w9m9f\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.636606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.636277 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-model-cache\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.636606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.636369 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-home\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.655330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.655287 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:33.719432 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.719339 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt"] Nov 03 19:57:33.735691 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.735577 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:33.748743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.739517 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:57:33.748743 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.740266 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7a62d4\"/\"test-model-7a62d4-kserve-self-signed-certs\"" Nov 03 19:57:33.749387 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.749358 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-tmp-dir\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.749465 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.749431 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-dshm\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.749770 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.749740 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9m9f\" (UniqueName: \"kubernetes.io/projected/f468d654-86d4-4e80-a134-3c566229a9f5-kube-api-access-w9m9f\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.749874 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.749806 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-model-cache\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.749933 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.749899 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-home\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.750021 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.750004 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468d654-86d4-4e80-a134-3c566229a9f5-tls-certs\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.766760 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.755241 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468d654-86d4-4e80-a134-3c566229a9f5-tls-certs\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.766760 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.755527 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-tmp-dir\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.771863 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.771839 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-model-cache\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.781190 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.772525 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-home\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.824306 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.818188 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9m9f\" (UniqueName: \"kubernetes.io/projected/f468d654-86d4-4e80-a134-3c566229a9f5-kube-api-access-w9m9f\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.888205 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.885920 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-dshm\") pod \"test-model-7a62d4-kserve-55f6d5c45f-kzqmt\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:33.942606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:33.939995 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:34.532009 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:34.531960 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt"] Nov 03 19:57:34.707450 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:34.707407 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" event={"ID":"f468d654-86d4-4e80-a134-3c566229a9f5","Type":"ContainerStarted","Data":"3c67cb5adaa7f7afafbb01e4e3ec94866fc2de457637cec45d2b3015b5351088"} Nov 03 19:57:35.734431 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:35.734391 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" event={"ID":"f468d654-86d4-4e80-a134-3c566229a9f5","Type":"ContainerStarted","Data":"b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965"} Nov 03 19:57:35.756228 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:35.756176 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" podStartSLOduration=3.7561625320000003 podStartE2EDuration="3.756162532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:35.752073433 +0000 UTC m=+1563.456912775" watchObservedRunningTime="2026-11-03 19:57:35.756162532 +0000 UTC m=+1563.461001872" Nov 03 19:57:35.944604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:35.942861 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:40.315119 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.315065 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9"] Nov 03 19:57:40.323247 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.323214 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.329653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.329617 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9"] Nov 03 19:57:40.488149 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488091 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02179854-1285-4182-aca7-1e083832067a-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488188 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02179854-1285-4182-aca7-1e083832067a-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488233 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488290 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488349 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488343 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488426 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488614 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488489 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488732 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488695 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.488791 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.488773 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll2sh\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-kube-api-access-ll2sh\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593067 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.592969 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593261 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593166 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593261 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593243 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll2sh\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-kube-api-access-ll2sh\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593393 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593366 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02179854-1285-4182-aca7-1e083832067a-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593418 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02179854-1285-4182-aca7-1e083832067a-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593501 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593441 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593501 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593457 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593784 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593806 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.593902 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593879 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.594010 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.593936 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.594065 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.594021 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.594405 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.594379 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.595250 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.595227 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02179854-1285-4182-aca7-1e083832067a-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.597303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.597272 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02179854-1285-4182-aca7-1e083832067a-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.597575 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.597555 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.603241 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.603217 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll2sh\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-kube-api-access-ll2sh\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.604130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.604101 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:40.684833 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:40.684512 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:41.115130 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:41.115100 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9"] Nov 03 19:57:41.118634 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:57:41.118601 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02179854_1285_4182_aca7_1e083832067a.slice/crio-6004dbe48cfb5b1145ebc5c9331bfda3c74128f158408d49ea5c0e7e7a78cf62 WatchSource:0}: Error finding container 6004dbe48cfb5b1145ebc5c9331bfda3c74128f158408d49ea5c0e7e7a78cf62: Status 404 returned error can't find the container with id 6004dbe48cfb5b1145ebc5c9331bfda3c74128f158408d49ea5c0e7e7a78cf62 Nov 03 19:57:41.122255 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:41.122206 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:41.122376 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:41.122302 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:41.122376 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:41.122344 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:41.839253 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:41.839211 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" event={"ID":"02179854-1285-4182-aca7-1e083832067a","Type":"ContainerStarted","Data":"1b930b64d1860f96ef932512107a8fa4a9740ba52fb3fcd5896c4d8609741932"} Nov 03 19:57:41.839747 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:41.839277 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" event={"ID":"02179854-1285-4182-aca7-1e083832067a","Type":"ContainerStarted","Data":"6004dbe48cfb5b1145ebc5c9331bfda3c74128f158408d49ea5c0e7e7a78cf62"} Nov 03 19:57:41.863955 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:41.863892 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podStartSLOduration=1.863874043 podStartE2EDuration="1.863874043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:41.863697094 +0000 UTC m=+1569.568536435" watchObservedRunningTime="2026-11-03 19:57:41.863874043 +0000 UTC m=+1569.568713385" Nov 03 19:57:42.693607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:42.692767 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:42.695297 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:42.695264 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:57:42.695437 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:42.695339 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 19:57:42.830778 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:42.830718 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:57:42.830986 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:42.830805 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:43.690121 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:43.690081 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:57:43.690884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:43.690838 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 19:57:43.953518 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:43.943026 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:44.827488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.009984 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:44.827488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.694145 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:57:44.827488 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.694202 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 19:57:44.911869 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.911838 4314 generic.go:356] "Generic (PLEG): container finished" podID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerID="2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202" exitCode=0 Nov 03 19:57:44.912188 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.912146 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" event={"ID":"3ac9f8e4-f04d-48af-8f72-d364a9402172","Type":"ContainerDied","Data":"2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202"} Nov 03 19:57:44.913316 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.913254 4314 scope.go:122] "RemoveContainer" containerID="2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202" Nov 03 19:57:44.915567 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.915532 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:44.915775 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.915754 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:44.915949 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.915931 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:44.928036 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:44.928013 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:57:45.696238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.695738 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:57:45.696238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.695804 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 19:57:45.710603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.706154 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": read tcp 10.134.0.2:59788->10.134.0.114:15021: read: connection reset by peer" start-of-body= Nov 03 19:57:45.710603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.706203 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": read tcp 10.134.0.2:59788->10.134.0.114:15021: read: connection reset by peer" Nov 03 19:57:45.941852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.941815 4314 generic.go:356] "Generic (PLEG): container finished" podID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerID="1c30e50cbf659931eb96c80e8fbfee4c08d3e6197971195b936edc7c2940643c" exitCode=0 Nov 03 19:57:45.942267 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.942029 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" event={"ID":"3c349a0a-69ad-4f14-ba0d-292c4a009b17","Type":"ContainerDied","Data":"1c30e50cbf659931eb96c80e8fbfee4c08d3e6197971195b936edc7c2940643c"} Nov 03 19:57:45.942532 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.942513 4314 scope.go:122] "RemoveContainer" containerID="1c30e50cbf659931eb96c80e8fbfee4c08d3e6197971195b936edc7c2940643c" Nov 03 19:57:45.944935 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.944894 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:45.945027 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.944995 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:45.945081 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.945049 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:57:45.946368 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:45.946326 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" event={"ID":"3ac9f8e4-f04d-48af-8f72-d364a9402172","Type":"ContainerStarted","Data":"901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3"} Nov 03 19:57:46.317312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:46.316909 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:57:46.317312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:46.317242 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:57:46.317312 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:46.317286 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 19:57:46.689867 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:46.689773 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:57:46.690118 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:46.690092 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 19:57:46.977661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:46.976723 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" event={"ID":"3c349a0a-69ad-4f14-ba0d-292c4a009b17","Type":"ContainerStarted","Data":"50fb4ba4b1c76a6cad6f1222e6e9d73862813b0ce39db1cef4e0a1f835aeca23"} Nov 03 19:57:47.020487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:46.996662 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:57:47.319604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:47.319469 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:57:47.319604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:47.319538 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 19:57:47.696656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:47.687697 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:57:47.696656 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:47.687778 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 19:57:48.031605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.031495 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:57:48.031605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.031560 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 19:57:48.436981 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.433215 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:57:48.643733 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.643689 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 19:57:48.644375 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.579199 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:57:48.675605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.579167 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:48.675605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.655248 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:57:48.716661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.715708 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 19:57:48.716661 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:48.715805 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 19:57:49.004238 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:49.004198 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:57:49.004616 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:49.004561 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 19:57:49.318660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:49.318147 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:57:49.318660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:49.318211 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 19:57:49.797605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:49.793914 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:49.797605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:49.795618 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:49.803990 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:49.803967 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 19:57:50.008604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:50.008309 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:57:50.008604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:50.008372 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 19:57:50.322040 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:50.321993 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:57:50.322639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:50.322607 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 19:57:50.999115 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:50.999064 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:57:50.999434 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:50.999388 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 19:57:51.323717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:51.320935 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:57:51.323717 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:51.320995 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 19:57:52.010672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:52.006002 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:57:52.010672 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:52.006065 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 19:57:52.049722 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:57:52.049684 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:57:52.322655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:52.322123 4314 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 19:57:52.322655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:52.322232 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 19:57:52.997313 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:52.997277 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:57:52.997930 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:52.997894 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 19:57:53.322178 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:53.322112 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:57:53.322626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:53.322575 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:57:53.323217 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:53.323196 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 19:57:53.999274 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:53.997924 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 19:57:53.999274 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:53.998019 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 19:57:54.632246 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.632207 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs"] Nov 03 19:57:54.638482 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.638453 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs"] Nov 03 19:57:54.638644 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.638564 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.644948 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.644922 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3cee6a\"/\"test-model-3cee6a-kserve-self-signed-certs\"" Nov 03 19:57:54.645132 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.645113 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3cee6a\"/\"default-dockercfg-wfs5t\"" Nov 03 19:57:54.645304 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.645280 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3cee6a\"/\"openshift-service-ca.crt\"" Nov 03 19:57:54.645378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.645338 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3cee6a\"/\"kube-root-ca.crt\"" Nov 03 19:57:54.742146 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.742111 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-tmp-dir\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.742324 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.742225 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-model-cache\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.742396 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.742378 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvf7\" (UniqueName: \"kubernetes.io/projected/01743f4d-03b7-411a-a125-8224f133fca1-kube-api-access-zfvf7\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.742483 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.742468 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-home\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.742723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.742704 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01743f4d-03b7-411a-a125-8224f133fca1-tls-certs\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.742799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.742750 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-dshm\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.880655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.847822 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01743f4d-03b7-411a-a125-8224f133fca1-tls-certs\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.880655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.847900 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-dshm\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.880655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.848033 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-tmp-dir\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.880655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.848106 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-model-cache\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.880655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.848264 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvf7\" (UniqueName: \"kubernetes.io/projected/01743f4d-03b7-411a-a125-8224f133fca1-kube-api-access-zfvf7\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.880655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.848349 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-home\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.880655 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.875365 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-tmp-dir\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.887993 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.887912 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-home\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.888499 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.888475 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-model-cache\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.942967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.942920 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-dshm\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.956485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.956444 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvf7\" (UniqueName: \"kubernetes.io/projected/01743f4d-03b7-411a-a125-8224f133fca1-kube-api-access-zfvf7\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.967278 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.967243 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01743f4d-03b7-411a-a125-8224f133fca1-tls-certs\") pod \"test-model-3cee6a-kserve-65d9d4dcf7-h84cs\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:54.987203 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:54.987169 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:57:55.018416 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:55.018191 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:57:55.021726 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:55.020654 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:57:55.027000 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:55.026976 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 19:57:55.373372 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:57:55.373329 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01743f4d_03b7_411a_a125_8224f133fca1.slice/crio-f94c58bc5326704285d3c4a7a391ac7bfb1c79f88981f817713cc11b2ac5a3bd WatchSource:0}: Error finding container f94c58bc5326704285d3c4a7a391ac7bfb1c79f88981f817713cc11b2ac5a3bd: Status 404 returned error can't find the container with id f94c58bc5326704285d3c4a7a391ac7bfb1c79f88981f817713cc11b2ac5a3bd Nov 03 19:57:55.376070 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:55.376038 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs"] Nov 03 19:57:56.333608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:56.333545 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" event={"ID":"01743f4d-03b7-411a-a125-8224f133fca1","Type":"ContainerStarted","Data":"4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6"} Nov 03 19:57:56.333608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:56.333605 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" event={"ID":"01743f4d-03b7-411a-a125-8224f133fca1","Type":"ContainerStarted","Data":"f94c58bc5326704285d3c4a7a391ac7bfb1c79f88981f817713cc11b2ac5a3bd"} Nov 03 19:57:56.354007 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:56.353962 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podStartSLOduration=2.35394976 podStartE2EDuration="2.35394976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:56.349050255 +0000 UTC m=+1584.053889597" watchObservedRunningTime="2026-11-03 19:57:56.35394976 +0000 UTC m=+1584.058789101" Nov 03 19:57:56.988351 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:57:56.988314 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:58:04.987934 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:04.987896 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:58:05.001448 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:05.001422 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:58:05.598936 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:05.598904 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:58:21.993077 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:58:21.993038 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:58:35.249030 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:35.248996 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 19:58:41.361607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:41.360710 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 03 19:58:41.601605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:41.596941 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:42.085645 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:42.068906 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:43.612611 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:43.595705 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded" Nov 03 19:58:43.707755 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:43.707712 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.121788 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.121721 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.238059 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.238016 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.295845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.295788 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.555818 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.551424 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.647754 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.647704 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.692606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.686570 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.772753 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.770766 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:58:44.773831 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.773801 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:58:44.805180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.805097 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:58:44.805500 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.805479 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:58:44.849733 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.849627 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:44.850996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.850962 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:45.079736 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:44.972711 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:45.226127 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:45.128669 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:45.227953 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:45.227927 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:58:45.301905 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:45.301649 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:45.352845 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:45.348273 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:58:45.424613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:45.421215 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:58:45.544151 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:45.544122 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:58:48.722038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:48.719709 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:58:48.722038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:48.719829 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:49.367674 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:49.367635 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9"] Nov 03 19:58:49.370378 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:49.370338 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" containerID="cri-o://6a5dd594d30c853dece4ad75554120f83c7e57e52f85202ed1c2aa7adde2823a" gracePeriod=30 Nov 03 19:58:52.010139 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:58:51.992840 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:58:52.806552 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.806520 4314 generic.go:356] "Generic (PLEG): container finished" podID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerID="e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47" exitCode=0 Nov 03 19:58:52.806773 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.806606 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" event={"ID":"c956c314-2fcc-4302-a6ac-11834c80cf9a","Type":"ContainerDied","Data":"e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47"} Nov 03 19:58:52.807154 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.807132 4314 scope.go:122] "RemoveContainer" containerID="e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47" Nov 03 19:58:52.808799 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.808770 4314 generic.go:356] "Generic (PLEG): container finished" podID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerID="c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d" exitCode=0 Nov 03 19:58:52.808920 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.808845 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" event={"ID":"7eb09d7b-edcd-4f5b-a297-b700f4b32b59","Type":"ContainerDied","Data":"c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d"} Nov 03 19:58:52.809290 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.809263 4314 scope.go:122] "RemoveContainer" containerID="c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d" Nov 03 19:58:52.809710 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.809677 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.809806 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.809773 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.809871 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.809825 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.811144 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.811117 4314 generic.go:356] "Generic (PLEG): container finished" podID="f5374374-e149-4721-beaa-b32f10b11ea9" containerID="849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41" exitCode=0 Nov 03 19:58:52.811273 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.811253 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" event={"ID":"f5374374-e149-4721-beaa-b32f10b11ea9","Type":"ContainerDied","Data":"849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41"} Nov 03 19:58:52.811701 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.811680 4314 scope.go:122] "RemoveContainer" containerID="849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41" Nov 03 19:58:52.811814 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.811787 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.811880 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.811863 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.811928 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.811916 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.815306 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.815274 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.815453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.815355 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.815453 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.815404 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:58:52.854898 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.854872 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:58:52.855007 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:52.854923 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:58:54.088610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.088103 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" event={"ID":"c956c314-2fcc-4302-a6ac-11834c80cf9a","Type":"ContainerStarted","Data":"8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84"} Nov 03 19:58:54.163605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.162881 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" event={"ID":"7eb09d7b-edcd-4f5b-a297-b700f4b32b59","Type":"ContainerStarted","Data":"60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb"} Nov 03 19:58:54.225605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.219179 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" event={"ID":"f5374374-e149-4721-beaa-b32f10b11ea9","Type":"ContainerStarted","Data":"1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547"} Nov 03 19:58:54.323019 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.310745 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:58:54.373303 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.373167 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:58:54.374038 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.374012 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:58:54.374225 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.374202 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:58:54.642782 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.642666 4314 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 19:58:54.643079 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.643046 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 19:58:54.833605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.735930 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:58:54.833605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:54.736001 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 19:58:55.251650 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.247866 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:55.335665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.333771 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:55.414629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.413298 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:58:55.414629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.413356 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:58:55.414629 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.414384 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 19:58:55.456099 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.456059 4314 generic.go:356] "Generic (PLEG): container finished" podID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerID="6a5dd594d30c853dece4ad75554120f83c7e57e52f85202ed1c2aa7adde2823a" exitCode=0 Nov 03 19:58:55.494605 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:58:55.471488 4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.051s" Nov 03 19:58:55.494605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.471536 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:58:55.494605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.471603 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" event={"ID":"93af09a3-1a47-43b8-982b-33a3df9d5283","Type":"ContainerDied","Data":"6a5dd594d30c853dece4ad75554120f83c7e57e52f85202ed1c2aa7adde2823a"} Nov 03 19:58:55.494605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.471649 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:58:55.533610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.533141 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:58:55.533610 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.533212 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 19:58:55.625683 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.597671 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:55.642765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.635512 4314 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 19:58:55.642765 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.635572 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 19:58:55.989272 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.989175 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:58:55.989780 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:55.989730 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 19:58:56.213719 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.191147 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:58:56.273699 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.260826 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:56.305723 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.296513 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:56.403509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.353166 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:58:56.403509 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.353221 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:58:56.433956 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.432791 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:56.434607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.434283 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:58:56.480639 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.480508 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:58:56.608665 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.607682 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:56.664758 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.649478 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.738672 4314 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.738736 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.741447 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-kube-api-access-sd5l4\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-kube-api-access-sd5l4\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.741501 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-data\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.741574 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-credential-socket\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.741643 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-podinfo\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.741723 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-token\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-token\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.741867 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.742000 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-certs\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.742062 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93af09a3-1a47-43b8-982b-33a3df9d5283-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93af09a3-1a47-43b8-982b-33a3df9d5283-istiod-ca-cert\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.747854 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.742201 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-socket\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:56.759093 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.759061 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-data" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:56.764820 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.759484 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-socket" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:57.005660 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:56.967660 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:57.090608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.069480 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:58:57.468446 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.468399 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 19:58:57.492734 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.174110 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": context deadline exceeded" Nov 03 19:58:57.494073 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.135319 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:57.494269 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.494249 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:57.494353 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.240438 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:58:57.494774 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.494755 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:58:57.494945 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.494927 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s"] Nov 03 19:58:57.603139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.132060 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-certs" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:57.603139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.143812 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93af09a3-1a47-43b8-982b-33a3df9d5283-istiod-ca-cert" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:57.603139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.176377 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-credential-socket" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:57.603139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.392656 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-podinfo" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:58:57.603139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.176668 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:58:57.603139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.176701 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:57.603139 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.265008 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:57.630365 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.612928 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:58:57.630365 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.612983 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:58:57.652330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.321610 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:58:57.652330 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.645043 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:58:57.683391 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.683351 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded" Nov 03 19:58:57.706613 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.689149 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:57.707022 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.706981 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93af09a3-1a47-43b8-982b-33a3df9d5283-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:57.707140 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.707128 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:57.707221 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.707210 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:57.711503 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.711474 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:58:57.711735 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.711702 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 19:58:57.754150 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.440864 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:57.755540 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.755515 4314 cpu_manager.go:414] "containerMap: removing container" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" Nov 03 19:58:57.755722 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.755708 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" Nov 03 19:58:57.756148 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.756133 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" containerName="istio-proxy" Nov 03 19:58:57.798607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.781719 4314 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 19:58:57.798607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.781810 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 19:58:57.836606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.807798 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-token" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:57.836606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.807903 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-kube-api-access-sd5l4" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "kube-api-access-sd5l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:57.836606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.822708 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:57.866607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.866170 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" Nov 03 19:58:57.891996 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.891915 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s"] Nov 03 19:58:57.905622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.904674 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9" event={"ID":"93af09a3-1a47-43b8-982b-33a3df9d5283","Type":"ContainerDied","Data":"842de7725d3633a90d449bbdf06b05fb13bb43729cadb78d6f96eca22b75859b"} Nov 03 19:58:57.905622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.904771 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:58:57.905622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.904823 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:58:57.905622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.904859 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:58:57.905622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.904892 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:58:57.905622 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.892142 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:57.906106 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.905887 4314 scope.go:122] "RemoveContainer" containerID="6a5dd594d30c853dece4ad75554120f83c7e57e52f85202ed1c2aa7adde2823a" Nov 03 19:58:57.930607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.929811 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\") pod \"93af09a3-1a47-43b8-982b-33a3df9d5283\" (UID: \"93af09a3-1a47-43b8-982b-33a3df9d5283\") " Nov 03 19:58:57.934605 ip-10-0-133-204 kubenswrapper[4314]: W1103 19:58:57.931062 4314 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/93af09a3-1a47-43b8-982b-33a3df9d5283/volumes/kubernetes.io~empty-dir/istio-envoy Nov 03 19:58:57.934605 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.931086 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy" pod "93af09a3-1a47-43b8-982b-33a3df9d5283" (UID: "93af09a3-1a47-43b8-982b-33a3df9d5283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:57.936229 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.936189 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd5l4\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-kube-api-access-sd5l4\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:57.959808 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:57.959782 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.007519 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93af09a3-1a47-43b8-982b-33a3df9d5283-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.007659 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.010956 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.011419 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.121127 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.121192 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.121256 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.121496 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.121532 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.122090 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4vl\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-kube-api-access-xj4vl\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.122264 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9521c02-9805-49dc-ace6-032e561bd5c6-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.212973 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.122459 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.299657 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.287023 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.376606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.355359 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.443694 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.400492 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4vl\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-kube-api-access-xj4vl\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444275 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9521c02-9805-49dc-ace6-032e561bd5c6-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444389 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444490 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444531 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444563 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444658 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444888 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.444925 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.445553 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-data\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.446626 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.446297 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9521c02-9805-49dc-ace6-032e561bd5c6-istiod-ca-cert\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.505248 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.447130 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-credential-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.623745 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.623696 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:58.671703 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.671647 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:58:58.688623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.683843 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 19:58:58.688623 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.684845 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:58:58.690124 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.646286 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-certs\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.690653 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.690635 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:58:58.690862 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.690835 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:58:58.691485 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.691464 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded" Nov 03 19:58:58.711641 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.711612 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-podinfo\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.717862 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.690062 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:58:58.745607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.647098 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-socket\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.745607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.647911 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-envoy\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:58.745607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.648645 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:58:58.745607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.732029 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:58:58.745607 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:58.734159 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-token\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:58:59.225633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.225186 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:58:59.225633 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.225247 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 19:58:59.388608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.375960 4314 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 19:58:59.388608 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.376036 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 19:58:59.622846 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.622808 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:58:59.623490 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.623455 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:58:59.732536 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.732483 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:00.129602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.755614 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 19:59:00.129602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.755689 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 19:59:00.129602 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.755723 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 03 19:59:00.137606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.759000 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:59:00.137606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.133931 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 19:59:00.137606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.967439 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:00.137606 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.135599 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6mwnz" podUID="900bc5dd-60bc-4597-9d8d-33c2551b5213" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.31:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:00.149604 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.145279 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": context deadline exceeded" Nov 03 19:59:00.163664 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.162310 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:59:00.192180 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.192034 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 19:59:00.310959 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.310924 4314 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 19:59:00.311201 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.311177 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 19:59:00.352603 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:58:59.945451 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4vl\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-kube-api-access-xj4vl\") pod \"e2e-inf-a-225668-openshift-default-765b499d89-qjf5s\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:59:00.366655 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:59:00.366544 4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93af09a3_1a47_43b8_982b_33a3df9d5283.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93af09a3_1a47_43b8_982b_33a3df9d5283.slice/crio-842de7725d3633a90d449bbdf06b05fb13bb43729cadb78d6f96eca22b75859b\": RecentStats: unable to find data in memory cache]" Nov 03 19:59:00.373619 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.373563 4314 patch_prober.go:29] interesting pod/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 19:59:00.373865 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.373841 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 19:59:00.406967 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.405504 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:59:00.414236 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.412812 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 19:59:00.429268 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:59:00.420478 4314 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.493s" Nov 03 19:59:00.500028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.499849 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 19:59:00.500028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.499917 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 19:59:00.500028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.499939 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9"] Nov 03 19:59:00.500028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.499980 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 19:59:00.500028 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.500012 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 19:59:00.500435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.500044 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 19:59:00.500435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.500075 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:59:00.500435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.500102 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 19:59:00.500435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.500130 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 19:59:00.500435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.500158 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 19:59:00.500435 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.500184 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 19:59:00.508675 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.508423 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 19:59:00.508977 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.508904 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 19:59:00.509487 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.509470 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 19:59:00.519237 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.518510 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-pf7m9"] Nov 03 19:59:00.650242 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.644218 4314 patch_prober.go:29] interesting pod/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 19:59:00.650242 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.644279 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 19:59:00.932451 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.930838 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:59:00.932451 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.930980 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:59:00.932451 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.931032 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 19:59:00.942025 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:00.941838 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s"] Nov 03 19:59:01.358850 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.358771 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:59:01.359384 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.359322 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:59:01.360171 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.360151 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 19:59:01.516206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.507742 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" event={"ID":"d9521c02-9805-49dc-ace6-032e561bd5c6","Type":"ContainerStarted","Data":"d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac"} Nov 03 19:59:01.516206 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.507823 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" event={"ID":"d9521c02-9805-49dc-ace6-032e561bd5c6","Type":"ContainerStarted","Data":"2ab8075f52a4ab0a9d4a82a0b7da041beec497fbc354d938cce85bda1c4e14b3"} Nov 03 19:59:01.525820 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.524500 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:59:01.526853 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.526828 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 19:59:01.545398 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.545344 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podStartSLOduration=81.545326925 podStartE2EDuration="1m21.545326925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:01.539368571 +0000 UTC m=+1649.244207912" watchObservedRunningTime="2026-11-03 19:59:01.545326925 +0000 UTC m=+1649.250166266" Nov 03 19:59:01.637852 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.637779 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:59:01.640227 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:01.640206 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 19:59:02.434829 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:02.434797 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:59:02.435286 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:02.434853 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 19:59:02.437202 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:02.437039 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93af09a3-1a47-43b8-982b-33a3df9d5283" path="/var/lib/kubelet/pods/93af09a3-1a47-43b8-982b-33a3df9d5283/volumes" Nov 03 19:59:02.438077 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:02.437473 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:59:03.433220 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:03.433184 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:59:03.433413 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:03.433247 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 19:59:04.433460 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:04.433424 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:59:04.433947 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:04.433485 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 19:59:05.433243 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:05.433213 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:59:05.433418 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:05.433263 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 19:59:06.433537 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:06.433491 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:59:06.433976 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:06.433558 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 19:59:07.433020 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:07.432981 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 03 19:59:07.433334 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:07.433065 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 03 19:59:08.438884 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:08.438854 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:59:08.439374 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:08.439353 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:59:08.440187 ip-10-0-133-204 kubenswrapper[4314]: I1103 19:59:08.440165 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 19:59:21.993114 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:59:21.993071 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 19:59:51.995030 ip-10-0-133-204 kubenswrapper[4314]: E1103 19:59:51.994988 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:00:00.140731 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.140691 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr"] Nov 03 20:00:00.145537 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.145507 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.146142 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.146116 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6"] Nov 03 20:00:00.153201 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.153177 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:00:00.154059 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.154034 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.156557 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.156534 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr"] Nov 03 20:00:00.225760 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.225718 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfq9\" (UniqueName: \"kubernetes.io/projected/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-kube-api-access-kcfq9\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.226216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.226175 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cb6f9-f30c-4f4f-b277-58be9487f1d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-ch2t6\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.226529 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.226505 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.226794 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.226772 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbvq\" (UniqueName: \"kubernetes.io/projected/573cb6f9-f30c-4f4f-b277-58be9487f1d3-kube-api-access-bbbvq\") pod \"maas-api-key-cleanup-29895600-ch2t6\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.240372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.240328 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6"] Nov 03 20:00:00.248248 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.248216 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc"] Nov 03 20:00:00.254893 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.254864 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.255328 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.255304 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl"] Nov 03 20:00:00.272604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.260863 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.272604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.267455 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc"] Nov 03 20:00:00.284632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.273283 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl"] Nov 03 20:00:00.284632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.278257 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn"] Nov 03 20:00:00.285154 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.285137 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.286130 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.286092 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb"] Nov 03 20:00:00.322080 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.321913 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn"] Nov 03 20:00:00.322243 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.322125 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb"] Nov 03 20:00:00.322243 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.322037 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:00.328021 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.327992 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.328273 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328094 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbvq\" (UniqueName: \"kubernetes.io/projected/573cb6f9-f30c-4f4f-b277-58be9487f1d3-kube-api-access-bbbvq\") pod \"maas-api-key-cleanup-29895600-ch2t6\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.328273 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328195 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c332276-3c4a-4e62-a917-f7070cb1ceff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.328408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328288 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfq9\" (UniqueName: \"kubernetes.io/projected/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-kube-api-access-kcfq9\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.328408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328346 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnlk\" (UniqueName: \"kubernetes.io/projected/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-kube-api-access-9pnlk\") pod \"maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:00.328520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328414 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwpr\" (UniqueName: \"kubernetes.io/projected/7c332276-3c4a-4e62-a917-f7070cb1ceff-kube-api-access-xhwpr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.328520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328440 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s7w\" (UniqueName: \"kubernetes.io/projected/7ee99a2b-22ad-49b8-bdd6-939337636258-kube-api-access-88s7w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.328520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328489 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee99a2b-22ad-49b8-bdd6-939337636258-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.328749 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328542 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79xv\" (UniqueName: \"kubernetes.io/projected/4a0115fa-f263-4cbe-94f6-4671979a6a6f-kube-api-access-c79xv\") pod \"maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.328811 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328742 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cb6f9-f30c-4f4f-b277-58be9487f1d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-ch2t6\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.328811 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328799 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0115fa-f263-4cbe-94f6-4671979a6a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.328916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328827 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:00.328916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.328860 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.329642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.329619 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cb6f9-f30c-4f4f-b277-58be9487f1d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895600-ch2t6\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.342523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.342490 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbvq\" (UniqueName: \"kubernetes.io/projected/573cb6f9-f30c-4f4f-b277-58be9487f1d3-kube-api-access-bbbvq\") pod \"maas-api-key-cleanup-29895600-ch2t6\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.342744 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.342726 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfq9\" (UniqueName: \"kubernetes.io/projected/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-kube-api-access-kcfq9\") pod \"maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.431062 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.430966 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0115fa-f263-4cbe-94f6-4671979a6a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.431062 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.431017 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:00.431304 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.431245 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c332276-3c4a-4e62-a917-f7070cb1ceff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.431368 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.431346 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnlk\" (UniqueName: \"kubernetes.io/projected/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-kube-api-access-9pnlk\") pod \"maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:00.431421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.431403 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwpr\" (UniqueName: \"kubernetes.io/projected/7c332276-3c4a-4e62-a917-f7070cb1ceff-kube-api-access-xhwpr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.431475 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.431419 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s7w\" (UniqueName: \"kubernetes.io/projected/7ee99a2b-22ad-49b8-bdd6-939337636258-kube-api-access-88s7w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.431475 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.431456 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee99a2b-22ad-49b8-bdd6-939337636258-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.431642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.431508 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79xv\" (UniqueName: \"kubernetes.io/projected/4a0115fa-f263-4cbe-94f6-4671979a6a6f-kube-api-access-c79xv\") pod \"maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.432885 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.432857 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee99a2b-22ad-49b8-bdd6-939337636258-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.433359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.433334 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0115fa-f263-4cbe-94f6-4671979a6a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.433695 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.433673 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:00.434072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.434055 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c332276-3c4a-4e62-a917-f7070cb1ceff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.441708 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.441148 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwpr\" (UniqueName: \"kubernetes.io/projected/7c332276-3c4a-4e62-a917-f7070cb1ceff-kube-api-access-xhwpr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.441708 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.441282 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s7w\" (UniqueName: \"kubernetes.io/projected/7ee99a2b-22ad-49b8-bdd6-939337636258-kube-api-access-88s7w\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.442235 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.442212 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnlk\" (UniqueName: \"kubernetes.io/projected/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-kube-api-access-9pnlk\") pod \"maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:00.442480 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.442458 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79xv\" (UniqueName: \"kubernetes.io/projected/4a0115fa-f263-4cbe-94f6-4671979a6a6f-kube-api-access-c79xv\") pod \"maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.538457 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.538428 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:00.549016 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.548994 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:00.576329 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.576300 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:00.622786 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.622517 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:00.625441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.625380 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:00.648665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:00.640394 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:01.033876 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.033848 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn"] Nov 03 20:00:01.034600 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:01.034553 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c332276_3c4a_4e62_a917_f7070cb1ceff.slice/crio-94fd29c1a0d0200785c2c094016ab0da3ee56befd0eebc2c02f182c299d528e1 WatchSource:0}: Error finding container 94fd29c1a0d0200785c2c094016ab0da3ee56befd0eebc2c02f182c299d528e1: Status 404 returned error can't find the container with id 94fd29c1a0d0200785c2c094016ab0da3ee56befd0eebc2c02f182c299d528e1 Nov 03 20:00:01.042813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.042785 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr"] Nov 03 20:00:01.043318 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:01.043289 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6cdd3a_7bfe_4c46_8d99_4ff711fa8fae.slice/crio-febbbb5bc9d0402fb5627a4fdd98b7c43a2976b416986c4bf1175e6c2a7f09f9 WatchSource:0}: Error finding container febbbb5bc9d0402fb5627a4fdd98b7c43a2976b416986c4bf1175e6c2a7f09f9: Status 404 returned error can't find the container with id febbbb5bc9d0402fb5627a4fdd98b7c43a2976b416986c4bf1175e6c2a7f09f9 Nov 03 20:00:01.075037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.074946 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6"] Nov 03 20:00:01.093453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.093425 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc"] Nov 03 20:00:01.101958 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:01.101928 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee99a2b_22ad_49b8_bdd6_939337636258.slice/crio-0544438a97d3a08ed458524126dba8b0d3465d97740431e83032d54a221a14f3 WatchSource:0}: Error finding container 0544438a97d3a08ed458524126dba8b0d3465d97740431e83032d54a221a14f3: Status 404 returned error can't find the container with id 0544438a97d3a08ed458524126dba8b0d3465d97740431e83032d54a221a14f3 Nov 03 20:00:01.221135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.221109 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl"] Nov 03 20:00:01.227181 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.227157 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb"] Nov 03 20:00:01.333557 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:01.331413 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cae6a2c_09ce_4228_bd9c_f6bb42e5b6b3.slice/crio-0445dd20c98db2728044ca02917370b03c59baed180cca5449dbe2c19c324fef WatchSource:0}: Error finding container 0445dd20c98db2728044ca02917370b03c59baed180cca5449dbe2c19c324fef: Status 404 returned error can't find the container with id 0445dd20c98db2728044ca02917370b03c59baed180cca5449dbe2c19c324fef Nov 03 20:00:01.382368 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.382331 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" event={"ID":"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae","Type":"ContainerStarted","Data":"febbbb5bc9d0402fb5627a4fdd98b7c43a2976b416986c4bf1175e6c2a7f09f9"} Nov 03 20:00:01.383943 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.383914 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" event={"ID":"7c332276-3c4a-4e62-a917-f7070cb1ceff","Type":"ContainerStarted","Data":"94fd29c1a0d0200785c2c094016ab0da3ee56befd0eebc2c02f182c299d528e1"} Nov 03 20:00:01.385379 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.385355 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" event={"ID":"7ee99a2b-22ad-49b8-bdd6-939337636258","Type":"ContainerStarted","Data":"0544438a97d3a08ed458524126dba8b0d3465d97740431e83032d54a221a14f3"} Nov 03 20:00:01.386827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.386785 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" event={"ID":"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3","Type":"ContainerStarted","Data":"0445dd20c98db2728044ca02917370b03c59baed180cca5449dbe2c19c324fef"} Nov 03 20:00:01.418227 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.418181 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" event={"ID":"4a0115fa-f263-4cbe-94f6-4671979a6a6f","Type":"ContainerStarted","Data":"f0be039f1c5cff59e36fd92e8356d3e5005d7086db3d68fdc1aab1b6df807e08"} Nov 03 20:00:01.420134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:01.420105 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" event={"ID":"573cb6f9-f30c-4f4f-b277-58be9487f1d3","Type":"ContainerStarted","Data":"e578c0aaa406913e34fedc685d52f53fec8181c22fdbc1217d9f38eb08b82f1c"} Nov 03 20:00:02.438400 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.438364 4314 generic.go:356] "Generic (PLEG): container finished" podID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerID="7fab0b704ba4e2da5d32cc57abdeaaf2498537deef0b93d02a9c7e1505d20a11" exitCode=60 Nov 03 20:00:02.439038 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.439017 4314 scope.go:122] "RemoveContainer" containerID="7fab0b704ba4e2da5d32cc57abdeaaf2498537deef0b93d02a9c7e1505d20a11" Nov 03 20:00:02.441882 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.441851 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" event={"ID":"7c332276-3c4a-4e62-a917-f7070cb1ceff","Type":"ContainerStarted","Data":"9f3623058e6178ab0822a7f78eb68beeeecfa3cd6679ca5a8145b7842ca07dc6"} Nov 03 20:00:02.441979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.441909 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" event={"ID":"7ee99a2b-22ad-49b8-bdd6-939337636258","Type":"ContainerDied","Data":"7fab0b704ba4e2da5d32cc57abdeaaf2498537deef0b93d02a9c7e1505d20a11"} Nov 03 20:00:02.441979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.441931 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" event={"ID":"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3","Type":"ContainerStarted","Data":"30cf473c5fa036208f7afd1cee423772e31f04b01ece8e2616915e5a7570d0f8"} Nov 03 20:00:02.442907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.442561 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" event={"ID":"4a0115fa-f263-4cbe-94f6-4671979a6a6f","Type":"ContainerStarted","Data":"584a5b8839e9b8273de346a1e0ff424ee8e21411cf57c0925510a450f68d5911"} Nov 03 20:00:02.444377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.444292 4314 generic.go:356] "Generic (PLEG): container finished" podID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerID="397fa752eef3097d95e49a69d7b2ead17025c2f1740d2725b92113c01254c4a0" exitCode=60 Nov 03 20:00:02.444377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.444368 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" event={"ID":"573cb6f9-f30c-4f4f-b277-58be9487f1d3","Type":"ContainerDied","Data":"397fa752eef3097d95e49a69d7b2ead17025c2f1740d2725b92113c01254c4a0"} Nov 03 20:00:02.444640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.444621 4314 scope.go:122] "RemoveContainer" containerID="397fa752eef3097d95e49a69d7b2ead17025c2f1740d2725b92113c01254c4a0" Nov 03 20:00:02.446306 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.446266 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerID="7bf5f806f8bdbca2e97f9fd377898ff3dc8863948e0b0f0c9f285041b645f7c2" exitCode=60 Nov 03 20:00:02.446432 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.446324 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" event={"ID":"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae","Type":"ContainerDied","Data":"7bf5f806f8bdbca2e97f9fd377898ff3dc8863948e0b0f0c9f285041b645f7c2"} Nov 03 20:00:02.446716 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.446612 4314 scope.go:122] "RemoveContainer" containerID="7bf5f806f8bdbca2e97f9fd377898ff3dc8863948e0b0f0c9f285041b645f7c2" Nov 03 20:00:02.455241 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.455195 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" podStartSLOduration=2.455181892 podStartE2EDuration="2.455181892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.452473493 +0000 UTC m=+1710.157312834" watchObservedRunningTime="2026-11-03 20:00:02.455181892 +0000 UTC m=+1710.160021231" Nov 03 20:00:02.523688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.523645 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" podStartSLOduration=2.523630385 podStartE2EDuration="2.523630385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.515848899 +0000 UTC m=+1710.220688242" watchObservedRunningTime="2026-11-03 20:00:02.523630385 +0000 UTC m=+1710.228469726" Nov 03 20:00:02.537574 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:02.537523 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" podStartSLOduration=2.537510301 podStartE2EDuration="2.537510301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.533007976 +0000 UTC m=+1710.237847317" watchObservedRunningTime="2026-11-03 20:00:02.537510301 +0000 UTC m=+1710.242349685" Nov 03 20:00:03.456956 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.456867 4314 generic.go:356] "Generic (PLEG): container finished" podID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerID="9f3623058e6178ab0822a7f78eb68beeeecfa3cd6679ca5a8145b7842ca07dc6" exitCode=60 Nov 03 20:00:03.457397 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.457022 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" event={"ID":"7c332276-3c4a-4e62-a917-f7070cb1ceff","Type":"ContainerDied","Data":"9f3623058e6178ab0822a7f78eb68beeeecfa3cd6679ca5a8145b7842ca07dc6"} Nov 03 20:00:03.457540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.457514 4314 scope.go:122] "RemoveContainer" containerID="9f3623058e6178ab0822a7f78eb68beeeecfa3cd6679ca5a8145b7842ca07dc6" Nov 03 20:00:03.462648 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.462623 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" event={"ID":"7ee99a2b-22ad-49b8-bdd6-939337636258","Type":"ContainerStarted","Data":"83338737a14f2b080e9064b75a79dbc93a9f6bfb5b43ac836b75f84dde795281"} Nov 03 20:00:03.477530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.477500 4314 generic.go:356] "Generic (PLEG): container finished" podID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerID="30cf473c5fa036208f7afd1cee423772e31f04b01ece8e2616915e5a7570d0f8" exitCode=60 Nov 03 20:00:03.477692 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.477645 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" event={"ID":"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3","Type":"ContainerDied","Data":"30cf473c5fa036208f7afd1cee423772e31f04b01ece8e2616915e5a7570d0f8"} Nov 03 20:00:03.478084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.478063 4314 scope.go:122] "RemoveContainer" containerID="30cf473c5fa036208f7afd1cee423772e31f04b01ece8e2616915e5a7570d0f8" Nov 03 20:00:03.488625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.488572 4314 generic.go:356] "Generic (PLEG): container finished" podID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerID="584a5b8839e9b8273de346a1e0ff424ee8e21411cf57c0925510a450f68d5911" exitCode=60 Nov 03 20:00:03.489047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.489024 4314 scope.go:122] "RemoveContainer" containerID="584a5b8839e9b8273de346a1e0ff424ee8e21411cf57c0925510a450f68d5911" Nov 03 20:00:03.489314 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.489286 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" event={"ID":"4a0115fa-f263-4cbe-94f6-4671979a6a6f","Type":"ContainerDied","Data":"584a5b8839e9b8273de346a1e0ff424ee8e21411cf57c0925510a450f68d5911"} Nov 03 20:00:03.491339 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.491310 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" event={"ID":"573cb6f9-f30c-4f4f-b277-58be9487f1d3","Type":"ContainerStarted","Data":"4d78f7fac513ed85df53c5c99d522a87f698d647c6a048da55a38b420897c579"} Nov 03 20:00:03.494810 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.494151 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" event={"ID":"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae","Type":"ContainerStarted","Data":"b2dfbe4f1beeadc522df22ca6e421aa8e45c9c294469e1a59b7f50bd5c6cf5c4"} Nov 03 20:00:03.498702 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.496956 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" podStartSLOduration=3.496940314 podStartE2EDuration="3.496940314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.493339871 +0000 UTC m=+1711.198179213" watchObservedRunningTime="2026-11-03 20:00:03.496940314 +0000 UTC m=+1711.201779655" Nov 03 20:00:03.528712 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.528666 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" podStartSLOduration=3.528653649 podStartE2EDuration="3.528653649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.522074308 +0000 UTC m=+1711.226913653" watchObservedRunningTime="2026-11-03 20:00:03.528653649 +0000 UTC m=+1711.233492988" Nov 03 20:00:03.586031 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:03.585975 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" podStartSLOduration=3.585957961 podStartE2EDuration="3.585957961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:03.576861762 +0000 UTC m=+1711.281701103" watchObservedRunningTime="2026-11-03 20:00:03.585957961 +0000 UTC m=+1711.290797302" Nov 03 20:00:04.519885 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.519851 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" event={"ID":"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3","Type":"ContainerStarted","Data":"d69e50b6b6b847f09e85309d10a482848f772a88decd7090a183877e3a1120bf"} Nov 03 20:00:04.531593 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.531545 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" event={"ID":"4a0115fa-f263-4cbe-94f6-4671979a6a6f","Type":"ContainerStarted","Data":"350d4fc51567b9fd21536d0b0d5d11e4d0332b6b18eba8072235a4e832cc5a7d"} Nov 03 20:00:04.535108 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.535083 4314 generic.go:356] "Generic (PLEG): container finished" podID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerID="4d78f7fac513ed85df53c5c99d522a87f698d647c6a048da55a38b420897c579" exitCode=60 Nov 03 20:00:04.535362 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.535346 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" event={"ID":"573cb6f9-f30c-4f4f-b277-58be9487f1d3","Type":"ContainerDied","Data":"4d78f7fac513ed85df53c5c99d522a87f698d647c6a048da55a38b420897c579"} Nov 03 20:00:04.535507 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.535495 4314 scope.go:122] "RemoveContainer" containerID="397fa752eef3097d95e49a69d7b2ead17025c2f1740d2725b92113c01254c4a0" Nov 03 20:00:04.536196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.536181 4314 scope.go:122] "RemoveContainer" containerID="4d78f7fac513ed85df53c5c99d522a87f698d647c6a048da55a38b420897c579" Nov 03 20:00:04.536846 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:04.536824 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-ch2t6_odh-ai-gateway-infra(573cb6f9-f30c-4f4f-b277-58be9487f1d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" Nov 03 20:00:04.541445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.541411 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerID="b2dfbe4f1beeadc522df22ca6e421aa8e45c9c294469e1a59b7f50bd5c6cf5c4" exitCode=60 Nov 03 20:00:04.550142 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.550124 4314 scope.go:122] "RemoveContainer" containerID="b2dfbe4f1beeadc522df22ca6e421aa8e45c9c294469e1a59b7f50bd5c6cf5c4" Nov 03 20:00:04.550756 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:04.550731 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr_odh-ai-gateway-infra(5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" Nov 03 20:00:04.571784 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.555572 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" event={"ID":"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae","Type":"ContainerDied","Data":"b2dfbe4f1beeadc522df22ca6e421aa8e45c9c294469e1a59b7f50bd5c6cf5c4"} Nov 03 20:00:04.572257 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.572229 4314 scope.go:122] "RemoveContainer" containerID="7bf5f806f8bdbca2e97f9fd377898ff3dc8863948e0b0f0c9f285041b645f7c2" Nov 03 20:00:04.573952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.573572 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" event={"ID":"7c332276-3c4a-4e62-a917-f7070cb1ceff","Type":"ContainerStarted","Data":"510388e5e5663c0ca15c09a2458d1bb900afd2e6871a7073c83515934953ef73"} Nov 03 20:00:04.578137 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.578112 4314 generic.go:356] "Generic (PLEG): container finished" podID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerID="83338737a14f2b080e9064b75a79dbc93a9f6bfb5b43ac836b75f84dde795281" exitCode=60 Nov 03 20:00:04.578259 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.578177 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" event={"ID":"7ee99a2b-22ad-49b8-bdd6-939337636258","Type":"ContainerDied","Data":"83338737a14f2b080e9064b75a79dbc93a9f6bfb5b43ac836b75f84dde795281"} Nov 03 20:00:04.578678 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.578658 4314 scope.go:122] "RemoveContainer" containerID="83338737a14f2b080e9064b75a79dbc93a9f6bfb5b43ac836b75f84dde795281" Nov 03 20:00:04.579059 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:04.579035 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc_odh-ai-gateway-infra(7ee99a2b-22ad-49b8-bdd6-939337636258)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" Nov 03 20:00:04.595660 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:04.595642 4314 scope.go:122] "RemoveContainer" containerID="7fab0b704ba4e2da5d32cc57abdeaaf2498537deef0b93d02a9c7e1505d20a11" Nov 03 20:00:05.594659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.594568 4314 scope.go:122] "RemoveContainer" containerID="4d78f7fac513ed85df53c5c99d522a87f698d647c6a048da55a38b420897c579" Nov 03 20:00:05.595183 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:05.595075 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-ch2t6_odh-ai-gateway-infra(573cb6f9-f30c-4f4f-b277-58be9487f1d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" Nov 03 20:00:05.596520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.596494 4314 scope.go:122] "RemoveContainer" containerID="b2dfbe4f1beeadc522df22ca6e421aa8e45c9c294469e1a59b7f50bd5c6cf5c4" Nov 03 20:00:05.596965 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:05.596941 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr_odh-ai-gateway-infra(5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" Nov 03 20:00:05.598495 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.598471 4314 generic.go:356] "Generic (PLEG): container finished" podID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerID="510388e5e5663c0ca15c09a2458d1bb900afd2e6871a7073c83515934953ef73" exitCode=60 Nov 03 20:00:05.598691 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.598668 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" event={"ID":"7c332276-3c4a-4e62-a917-f7070cb1ceff","Type":"ContainerDied","Data":"510388e5e5663c0ca15c09a2458d1bb900afd2e6871a7073c83515934953ef73"} Nov 03 20:00:05.598833 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.598721 4314 scope.go:122] "RemoveContainer" containerID="9f3623058e6178ab0822a7f78eb68beeeecfa3cd6679ca5a8145b7842ca07dc6" Nov 03 20:00:05.599052 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.599024 4314 scope.go:122] "RemoveContainer" containerID="510388e5e5663c0ca15c09a2458d1bb900afd2e6871a7073c83515934953ef73" Nov 03 20:00:05.599534 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:05.599507 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn_odh-ai-gateway-infra(7c332276-3c4a-4e62-a917-f7070cb1ceff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" Nov 03 20:00:05.600914 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.600899 4314 scope.go:122] "RemoveContainer" containerID="83338737a14f2b080e9064b75a79dbc93a9f6bfb5b43ac836b75f84dde795281" Nov 03 20:00:05.601274 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:05.601256 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc_odh-ai-gateway-infra(7ee99a2b-22ad-49b8-bdd6-939337636258)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" Nov 03 20:00:05.602848 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.602805 4314 generic.go:356] "Generic (PLEG): container finished" podID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerID="d69e50b6b6b847f09e85309d10a482848f772a88decd7090a183877e3a1120bf" exitCode=60 Nov 03 20:00:05.602953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.602862 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" event={"ID":"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3","Type":"ContainerDied","Data":"d69e50b6b6b847f09e85309d10a482848f772a88decd7090a183877e3a1120bf"} Nov 03 20:00:05.603173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.603151 4314 scope.go:122] "RemoveContainer" containerID="d69e50b6b6b847f09e85309d10a482848f772a88decd7090a183877e3a1120bf" Nov 03 20:00:05.603518 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:05.603495 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb_odh-ai-gateway-infra(5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" Nov 03 20:00:05.605410 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.605388 4314 generic.go:356] "Generic (PLEG): container finished" podID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerID="350d4fc51567b9fd21536d0b0d5d11e4d0332b6b18eba8072235a4e832cc5a7d" exitCode=60 Nov 03 20:00:05.605509 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.605466 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" event={"ID":"4a0115fa-f263-4cbe-94f6-4671979a6a6f","Type":"ContainerDied","Data":"350d4fc51567b9fd21536d0b0d5d11e4d0332b6b18eba8072235a4e832cc5a7d"} Nov 03 20:00:05.605761 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.605743 4314 scope.go:122] "RemoveContainer" containerID="350d4fc51567b9fd21536d0b0d5d11e4d0332b6b18eba8072235a4e832cc5a7d" Nov 03 20:00:05.606119 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:05.606098 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl_odh-ai-gateway-infra(4a0115fa-f263-4cbe-94f6-4671979a6a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" Nov 03 20:00:05.615219 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.615193 4314 scope.go:122] "RemoveContainer" containerID="30cf473c5fa036208f7afd1cee423772e31f04b01ece8e2616915e5a7570d0f8" Nov 03 20:00:05.629751 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:05.629732 4314 scope.go:122] "RemoveContainer" containerID="584a5b8839e9b8273de346a1e0ff424ee8e21411cf57c0925510a450f68d5911" Nov 03 20:00:06.615989 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:06.615957 4314 scope.go:122] "RemoveContainer" containerID="510388e5e5663c0ca15c09a2458d1bb900afd2e6871a7073c83515934953ef73" Nov 03 20:00:06.616448 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:06.616425 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn_odh-ai-gateway-infra(7c332276-3c4a-4e62-a917-f7070cb1ceff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" Nov 03 20:00:06.617773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:06.617744 4314 scope.go:122] "RemoveContainer" containerID="d69e50b6b6b847f09e85309d10a482848f772a88decd7090a183877e3a1120bf" Nov 03 20:00:06.618187 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:06.618165 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb_odh-ai-gateway-infra(5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" Nov 03 20:00:06.619540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:06.619519 4314 scope.go:122] "RemoveContainer" containerID="350d4fc51567b9fd21536d0b0d5d11e4d0332b6b18eba8072235a4e832cc5a7d" Nov 03 20:00:06.619905 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:06.619886 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl_odh-ai-gateway-infra(4a0115fa-f263-4cbe-94f6-4671979a6a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" Nov 03 20:00:09.160832 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:09.160796 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:00:15.420833 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:15.420794 4314 scope.go:122] "RemoveContainer" containerID="b2dfbe4f1beeadc522df22ca6e421aa8e45c9c294469e1a59b7f50bd5c6cf5c4" Nov 03 20:00:16.420737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.420695 4314 scope.go:122] "RemoveContainer" containerID="83338737a14f2b080e9064b75a79dbc93a9f6bfb5b43ac836b75f84dde795281" Nov 03 20:00:16.420945 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.420885 4314 scope.go:122] "RemoveContainer" containerID="510388e5e5663c0ca15c09a2458d1bb900afd2e6871a7073c83515934953ef73" Nov 03 20:00:16.422547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.421698 4314 scope.go:122] "RemoveContainer" containerID="d69e50b6b6b847f09e85309d10a482848f772a88decd7090a183877e3a1120bf" Nov 03 20:00:16.422547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.421825 4314 scope.go:122] "RemoveContainer" containerID="350d4fc51567b9fd21536d0b0d5d11e4d0332b6b18eba8072235a4e832cc5a7d" Nov 03 20:00:16.841890 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.841858 4314 generic.go:356] "Generic (PLEG): container finished" podID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerID="4efffe8eec7c0d9554d9fb1079224ea36d608e67ecfdc0c88f56b708136412cd" exitCode=60 Nov 03 20:00:16.842053 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.841907 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" event={"ID":"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae","Type":"ContainerDied","Data":"4efffe8eec7c0d9554d9fb1079224ea36d608e67ecfdc0c88f56b708136412cd"} Nov 03 20:00:16.842053 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.841991 4314 scope.go:122] "RemoveContainer" containerID="b2dfbe4f1beeadc522df22ca6e421aa8e45c9c294469e1a59b7f50bd5c6cf5c4" Nov 03 20:00:16.842367 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:16.842344 4314 scope.go:122] "RemoveContainer" containerID="4efffe8eec7c0d9554d9fb1079224ea36d608e67ecfdc0c88f56b708136412cd" Nov 03 20:00:16.842760 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:16.842738 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr_odh-ai-gateway-infra(5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" Nov 03 20:00:17.421092 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.421056 4314 scope.go:122] "RemoveContainer" containerID="4d78f7fac513ed85df53c5c99d522a87f698d647c6a048da55a38b420897c579" Nov 03 20:00:17.852460 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.852429 4314 generic.go:356] "Generic (PLEG): container finished" podID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerID="10ca59bc69a3b3ab73e7cbce93f67abe2e6fd5143cf0f13d3e6c9cbf997c954b" exitCode=60 Nov 03 20:00:17.852639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.852498 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" event={"ID":"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3","Type":"ContainerDied","Data":"10ca59bc69a3b3ab73e7cbce93f67abe2e6fd5143cf0f13d3e6c9cbf997c954b"} Nov 03 20:00:17.852639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.852562 4314 scope.go:122] "RemoveContainer" containerID="d69e50b6b6b847f09e85309d10a482848f772a88decd7090a183877e3a1120bf" Nov 03 20:00:17.852958 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.852942 4314 scope.go:122] "RemoveContainer" containerID="10ca59bc69a3b3ab73e7cbce93f67abe2e6fd5143cf0f13d3e6c9cbf997c954b" Nov 03 20:00:17.853295 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:17.853273 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb_odh-ai-gateway-infra(5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" Nov 03 20:00:17.855066 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.855042 4314 generic.go:356] "Generic (PLEG): container finished" podID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerID="aeca103644936a87ebc139df13660231150eb0a4d4f648297541fdb6d95d6d75" exitCode=60 Nov 03 20:00:17.855180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.855069 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" event={"ID":"4a0115fa-f263-4cbe-94f6-4671979a6a6f","Type":"ContainerDied","Data":"aeca103644936a87ebc139df13660231150eb0a4d4f648297541fdb6d95d6d75"} Nov 03 20:00:17.855462 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.855443 4314 scope.go:122] "RemoveContainer" containerID="aeca103644936a87ebc139df13660231150eb0a4d4f648297541fdb6d95d6d75" Nov 03 20:00:17.855791 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:17.855764 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl_odh-ai-gateway-infra(4a0115fa-f263-4cbe-94f6-4671979a6a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" Nov 03 20:00:17.859376 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.859352 4314 generic.go:356] "Generic (PLEG): container finished" podID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerID="3cc7df3bb130ac3348b9ed84a56ccfd060f639b5644328d3efcef5f490797c03" exitCode=60 Nov 03 20:00:17.859493 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.859424 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" event={"ID":"7c332276-3c4a-4e62-a917-f7070cb1ceff","Type":"ContainerDied","Data":"3cc7df3bb130ac3348b9ed84a56ccfd060f639b5644328d3efcef5f490797c03"} Nov 03 20:00:17.859873 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.859851 4314 scope.go:122] "RemoveContainer" containerID="3cc7df3bb130ac3348b9ed84a56ccfd060f639b5644328d3efcef5f490797c03" Nov 03 20:00:17.860182 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:17.860144 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn_odh-ai-gateway-infra(7c332276-3c4a-4e62-a917-f7070cb1ceff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" Nov 03 20:00:17.861688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.861665 4314 generic.go:356] "Generic (PLEG): container finished" podID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerID="76875c9df1201698546dedb0d9e517ef907b6b72a3b4b9939552b7725bb15e38" exitCode=60 Nov 03 20:00:17.861758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.861730 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" event={"ID":"7ee99a2b-22ad-49b8-bdd6-939337636258","Type":"ContainerDied","Data":"76875c9df1201698546dedb0d9e517ef907b6b72a3b4b9939552b7725bb15e38"} Nov 03 20:00:17.862039 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.862025 4314 scope.go:122] "RemoveContainer" containerID="76875c9df1201698546dedb0d9e517ef907b6b72a3b4b9939552b7725bb15e38" Nov 03 20:00:17.862289 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:17.862273 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc_odh-ai-gateway-infra(7ee99a2b-22ad-49b8-bdd6-939337636258)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" Nov 03 20:00:17.873339 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.873319 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr"] Nov 03 20:00:17.913355 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.913332 4314 scope.go:122] "RemoveContainer" containerID="350d4fc51567b9fd21536d0b0d5d11e4d0332b6b18eba8072235a4e832cc5a7d" Nov 03 20:00:17.930846 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.930821 4314 scope.go:122] "RemoveContainer" containerID="510388e5e5663c0ca15c09a2458d1bb900afd2e6871a7073c83515934953ef73" Nov 03 20:00:17.948341 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:17.948314 4314 scope.go:122] "RemoveContainer" containerID="83338737a14f2b080e9064b75a79dbc93a9f6bfb5b43ac836b75f84dde795281" Nov 03 20:00:18.108849 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.108823 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:18.142886 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.142854 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j"] Nov 03 20:00:18.144973 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.144911 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.145123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.144975 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.145123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.145029 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.145123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.145038 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.145123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.145102 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.145123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.145110 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.145413 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.145394 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.145470 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.145423 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:18.170061 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.170031 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j"] Nov 03 20:00:18.170227 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.170145 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.177517 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.177487 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-kube-api-access-kcfq9\" (UniqueName: \"kubernetes.io/projected/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-kube-api-access-kcfq9\") pod \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " Nov 03 20:00:18.177707 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.177675 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-maas-api-service-ca\") pod \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\" (UID: \"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae\") " Nov 03 20:00:18.178014 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.177991 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-maas-api-service-ca" pod "5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" (UID: "5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:18.178871 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.178848 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:18.180408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.180385 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-kube-api-access-kcfq9" pod "5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" (UID: "5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae"). InnerVolumeSpecName "kube-api-access-kcfq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:18.282936 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.280513 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.282936 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.280735 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvk9\" (UniqueName: \"kubernetes.io/projected/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-kube-api-access-plvk9\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.282936 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.280966 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcfq9\" (UniqueName: \"kubernetes.io/projected/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae-kube-api-access-kcfq9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:18.383136 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.383099 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.383316 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.383231 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvk9\" (UniqueName: \"kubernetes.io/projected/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-kube-api-access-plvk9\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.383787 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.383766 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.391838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.391807 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvk9\" (UniqueName: \"kubernetes.io/projected/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-kube-api-access-plvk9\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.494666 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.494637 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:18.798033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.798005 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j"] Nov 03 20:00:18.799595 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:18.799558 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4288e8ce_c0ad_47c6_bb22_4648dfc617ea.slice/crio-f04ddc64114ac0c3251afb253e35fbe8e288d539ef3f47d2d7c538b1f5abf6f3 WatchSource:0}: Error finding container f04ddc64114ac0c3251afb253e35fbe8e288d539ef3f47d2d7c538b1f5abf6f3: Status 404 returned error can't find the container with id f04ddc64114ac0c3251afb253e35fbe8e288d539ef3f47d2d7c538b1f5abf6f3 Nov 03 20:00:18.876182 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.876145 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerStarted","Data":"f04ddc64114ac0c3251afb253e35fbe8e288d539ef3f47d2d7c538b1f5abf6f3"} Nov 03 20:00:18.885187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.885155 4314 generic.go:356] "Generic (PLEG): container finished" podID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerID="3590bdba38c1cc1d394858f8be0d3546d8c5218bd6fd3230e0c5843b092b8639" exitCode=60 Nov 03 20:00:18.885310 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.885221 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" event={"ID":"573cb6f9-f30c-4f4f-b277-58be9487f1d3","Type":"ContainerDied","Data":"3590bdba38c1cc1d394858f8be0d3546d8c5218bd6fd3230e0c5843b092b8639"} Nov 03 20:00:18.885310 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.885280 4314 scope.go:122] "RemoveContainer" containerID="4d78f7fac513ed85df53c5c99d522a87f698d647c6a048da55a38b420897c579" Nov 03 20:00:18.886075 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.885816 4314 scope.go:122] "RemoveContainer" containerID="3590bdba38c1cc1d394858f8be0d3546d8c5218bd6fd3230e0c5843b092b8639" Nov 03 20:00:18.886506 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:18.886481 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895600-ch2t6_odh-ai-gateway-infra(573cb6f9-f30c-4f4f-b277-58be9487f1d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" Nov 03 20:00:18.889859 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.889812 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" event={"ID":"5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae","Type":"ContainerDied","Data":"febbbb5bc9d0402fb5627a4fdd98b7c43a2976b416986c4bf1175e6c2a7f09f9"} Nov 03 20:00:18.890219 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.890073 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr" Nov 03 20:00:18.890306 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.890263 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb"] Nov 03 20:00:18.931507 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.931481 4314 scope.go:122] "RemoveContainer" containerID="4efffe8eec7c0d9554d9fb1079224ea36d608e67ecfdc0c88f56b708136412cd" Nov 03 20:00:18.932532 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:18.932498 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc"] Nov 03 20:00:19.077928 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.077835 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn"] Nov 03 20:00:19.123931 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.123775 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl"] Nov 03 20:00:19.168305 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.168260 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr"] Nov 03 20:00:19.177161 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.176096 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39043d-29895600-27rnr"] Nov 03 20:00:19.295424 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.295394 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:19.343082 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.343046 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg"] Nov 03 20:00:19.344896 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.344871 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.347979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.344897 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.347979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.344994 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.347979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.345004 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.347979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.345326 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" containerName="cleanup" Nov 03 20:00:19.347979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.345352 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.347979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.345368 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.354843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.352658 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.354843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.354083 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg"] Nov 03 20:00:19.426602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.426403 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:19.430803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.429349 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-kube-api-access-9pnlk\" (UniqueName: \"kubernetes.io/projected/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-kube-api-access-9pnlk\") pod \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " Nov 03 20:00:19.430803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.429414 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-maas-api-service-ca\") pod \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\" (UID: \"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3\") " Nov 03 20:00:19.430803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.430079 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv5cw\" (UniqueName: \"kubernetes.io/projected/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-kube-api-access-dv5cw\") pod \"maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.430803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.430230 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.432341 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.432317 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-maas-api-service-ca" pod "5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" (UID: "5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:19.433960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.433934 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-kube-api-access-9pnlk" pod "5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" (UID: "5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3"). InnerVolumeSpecName "kube-api-access-9pnlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:19.473035 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.472695 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.477684 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj"] Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479038 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479060 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479092 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479100 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479141 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479149 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479165 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479173 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479232 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479239 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479434 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479463 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479474 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479487 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479500 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.481952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.479517 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.484487 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.484346 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.504402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.504374 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj"] Nov 03 20:00:19.506891 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.506870 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:19.532802 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.532766 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c332276-3c4a-4e62-a917-f7070cb1ceff-kube-api-access-xhwpr\" (UniqueName: \"kubernetes.io/projected/7c332276-3c4a-4e62-a917-f7070cb1ceff-kube-api-access-xhwpr\") pod \"7c332276-3c4a-4e62-a917-f7070cb1ceff\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " Nov 03 20:00:19.533020 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.532842 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0115fa-f263-4cbe-94f6-4671979a6a6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0115fa-f263-4cbe-94f6-4671979a6a6f-maas-api-service-ca\") pod \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " Nov 03 20:00:19.533337 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.533309 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c332276-3c4a-4e62-a917-f7070cb1ceff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c332276-3c4a-4e62-a917-f7070cb1ceff-maas-api-service-ca\") pod \"7c332276-3c4a-4e62-a917-f7070cb1ceff\" (UID: \"7c332276-3c4a-4e62-a917-f7070cb1ceff\") " Nov 03 20:00:19.533403 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.533360 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0115fa-f263-4cbe-94f6-4671979a6a6f-kube-api-access-c79xv\" (UniqueName: \"kubernetes.io/projected/4a0115fa-f263-4cbe-94f6-4671979a6a6f-kube-api-access-c79xv\") pod \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\" (UID: \"4a0115fa-f263-4cbe-94f6-4671979a6a6f\") " Nov 03 20:00:19.533841 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.533816 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2691ff8-da25-4a4f-aec1-f748b2d097fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.534526 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.534497 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88krx\" (UniqueName: \"kubernetes.io/projected/d2691ff8-da25-4a4f-aec1-f748b2d097fa-kube-api-access-88krx\") pod \"maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.534341 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0115fa-f263-4cbe-94f6-4671979a6a6f-maas-api-service-ca" pod "4a0115fa-f263-4cbe-94f6-4671979a6a6f" (UID: "4a0115fa-f263-4cbe-94f6-4671979a6a6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.534825 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv5cw\" (UniqueName: \"kubernetes.io/projected/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-kube-api-access-dv5cw\") pod \"maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.535317 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.535415 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c332276-3c4a-4e62-a917-f7070cb1ceff-maas-api-service-ca" pod "7c332276-3c4a-4e62-a917-f7070cb1ceff" (UID: "7c332276-3c4a-4e62-a917-f7070cb1ceff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.535845 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c332276-3c4a-4e62-a917-f7070cb1ceff-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.535870 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0115fa-f263-4cbe-94f6-4671979a6a6f-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.535887 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pnlk\" (UniqueName: \"kubernetes.io/projected/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-kube-api-access-9pnlk\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.535902 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.535919 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.537804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.537233 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz"] Nov 03 20:00:19.538601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538562 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.538705 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538602 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.538705 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538661 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.538705 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538671 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.538705 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538688 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.538705 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538696 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" containerName="cleanup" Nov 03 20:00:19.538966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538758 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.538966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538766 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.538966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538783 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.538966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538791 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.539156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538981 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.539156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.538998 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.539156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.539021 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" containerName="cleanup" Nov 03 20:00:19.544262 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.544066 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.545359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.545090 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c332276-3c4a-4e62-a917-f7070cb1ceff-kube-api-access-xhwpr" pod "7c332276-3c4a-4e62-a917-f7070cb1ceff" (UID: "7c332276-3c4a-4e62-a917-f7070cb1ceff"). InnerVolumeSpecName "kube-api-access-xhwpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:19.573059 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.570013 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz"] Nov 03 20:00:19.575079 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.574738 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv5cw\" (UniqueName: \"kubernetes.io/projected/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-kube-api-access-dv5cw\") pod \"maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.579798 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.579727 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0115fa-f263-4cbe-94f6-4671979a6a6f-kube-api-access-c79xv" pod "4a0115fa-f263-4cbe-94f6-4671979a6a6f" (UID: "4a0115fa-f263-4cbe-94f6-4671979a6a6f"). InnerVolumeSpecName "kube-api-access-c79xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:19.606294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.606269 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq"] Nov 03 20:00:19.609814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.609788 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" containerName="cleanup" Nov 03 20:00:19.621370 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.621348 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq"] Nov 03 20:00:19.621804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.621735 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:19.637187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.637164 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee99a2b-22ad-49b8-bdd6-939337636258-kube-api-access-88s7w\" (UniqueName: \"kubernetes.io/projected/7ee99a2b-22ad-49b8-bdd6-939337636258-kube-api-access-88s7w\") pod \"7ee99a2b-22ad-49b8-bdd6-939337636258\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " Nov 03 20:00:19.637374 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.637360 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee99a2b-22ad-49b8-bdd6-939337636258-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee99a2b-22ad-49b8-bdd6-939337636258-maas-api-service-ca\") pod \"7ee99a2b-22ad-49b8-bdd6-939337636258\" (UID: \"7ee99a2b-22ad-49b8-bdd6-939337636258\") " Nov 03 20:00:19.637805 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.637786 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2691ff8-da25-4a4f-aec1-f748b2d097fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.637851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.637817 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88krx\" (UniqueName: \"kubernetes.io/projected/d2691ff8-da25-4a4f-aec1-f748b2d097fa-kube-api-access-88krx\") pod \"maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.637851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.637837 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fjv\" (UniqueName: \"kubernetes.io/projected/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-kube-api-access-d6fjv\") pod \"maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.638216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.638192 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.638377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.638359 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79xv\" (UniqueName: \"kubernetes.io/projected/4a0115fa-f263-4cbe-94f6-4671979a6a6f-kube-api-access-c79xv\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.638727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.638705 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwpr\" (UniqueName: \"kubernetes.io/projected/7c332276-3c4a-4e62-a917-f7070cb1ceff-kube-api-access-xhwpr\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.640607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.639870 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee99a2b-22ad-49b8-bdd6-939337636258-maas-api-service-ca" pod "7ee99a2b-22ad-49b8-bdd6-939337636258" (UID: "7ee99a2b-22ad-49b8-bdd6-939337636258"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:19.640607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.640544 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2691ff8-da25-4a4f-aec1-f748b2d097fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.648551 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.648502 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88krx\" (UniqueName: \"kubernetes.io/projected/d2691ff8-da25-4a4f-aec1-f748b2d097fa-kube-api-access-88krx\") pod \"maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.653274 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.650699 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee99a2b-22ad-49b8-bdd6-939337636258-kube-api-access-88s7w" pod "7ee99a2b-22ad-49b8-bdd6-939337636258" (UID: "7ee99a2b-22ad-49b8-bdd6-939337636258"). InnerVolumeSpecName "kube-api-access-88s7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:19.684737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.684487 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:19.740537 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.740453 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fjv\" (UniqueName: \"kubernetes.io/projected/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-kube-api-access-d6fjv\") pod \"maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.741630 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.740500 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:19.741630 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.740961 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngk5k\" (UniqueName: \"kubernetes.io/projected/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-kube-api-access-ngk5k\") pod \"maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:19.741630 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.740984 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.741630 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.741092 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s7w\" (UniqueName: \"kubernetes.io/projected/7ee99a2b-22ad-49b8-bdd6-939337636258-kube-api-access-88s7w\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.741630 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.741105 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee99a2b-22ad-49b8-bdd6-939337636258-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:19.741630 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.741617 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.750368 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.750341 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fjv\" (UniqueName: \"kubernetes.io/projected/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-kube-api-access-d6fjv\") pod \"maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.811033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.810999 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:19.843171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.843135 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk5k\" (UniqueName: \"kubernetes.io/projected/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-kube-api-access-ngk5k\") pod \"maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:19.843430 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.843407 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:19.844166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.844139 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:19.855740 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.855717 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngk5k\" (UniqueName: \"kubernetes.io/projected/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-kube-api-access-ngk5k\") pod \"maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:19.904557 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.904430 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:19.919813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.915953 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerStarted","Data":"1905cbe4e59a37eb45dcf528c3ea7404bcd16e5c7bfa1619eb93b8d3d4be5287"} Nov 03 20:00:19.927675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.927282 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" event={"ID":"7ee99a2b-22ad-49b8-bdd6-939337636258","Type":"ContainerDied","Data":"0544438a97d3a08ed458524126dba8b0d3465d97740431e83032d54a221a14f3"} Nov 03 20:00:19.929779 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.927854 4314 scope.go:122] "RemoveContainer" containerID="76875c9df1201698546dedb0d9e517ef907b6b72a3b4b9939552b7725bb15e38" Nov 03 20:00:19.929779 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.927990 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc" Nov 03 20:00:19.933676 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.933648 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6"] Nov 03 20:00:19.939283 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.938172 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" event={"ID":"5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3","Type":"ContainerDied","Data":"0445dd20c98db2728044ca02917370b03c59baed180cca5449dbe2c19c324fef"} Nov 03 20:00:19.939283 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.938272 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb" Nov 03 20:00:19.944791 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.944518 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" event={"ID":"4a0115fa-f263-4cbe-94f6-4671979a6a6f","Type":"ContainerDied","Data":"f0be039f1c5cff59e36fd92e8356d3e5005d7086db3d68fdc1aab1b6df807e08"} Nov 03 20:00:19.944791 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.944628 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl" Nov 03 20:00:19.951169 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.950929 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" podStartSLOduration=19.95091803 podStartE2EDuration="19.95091803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:19.942421842 +0000 UTC m=+1727.647261182" watchObservedRunningTime="2026-11-03 20:00:19.95091803 +0000 UTC m=+1727.655757415" Nov 03 20:00:19.954841 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.954793 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" event={"ID":"7c332276-3c4a-4e62-a917-f7070cb1ceff","Type":"ContainerDied","Data":"94fd29c1a0d0200785c2c094016ab0da3ee56befd0eebc2c02f182c299d528e1"} Nov 03 20:00:19.955168 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.955102 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn" Nov 03 20:00:19.990263 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:19.989403 4314 scope.go:122] "RemoveContainer" containerID="10ca59bc69a3b3ab73e7cbce93f67abe2e6fd5143cf0f13d3e6c9cbf997c954b" Nov 03 20:00:20.000382 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:20.000331 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405b9d8b_90d1_4e78_aa4b_efc2dd910cff.slice/crio-e462ecbf5c629d48a06cc8719dc881c36abbab8c67e74755e9f5cd168e33c26a WatchSource:0}: Error finding container e462ecbf5c629d48a06cc8719dc881c36abbab8c67e74755e9f5cd168e33c26a: Status 404 returned error can't find the container with id e462ecbf5c629d48a06cc8719dc881c36abbab8c67e74755e9f5cd168e33c26a Nov 03 20:00:20.004388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.004197 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:20.033176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.033143 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg"] Nov 03 20:00:20.049278 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.049255 4314 scope.go:122] "RemoveContainer" containerID="aeca103644936a87ebc139df13660231150eb0a4d4f648297541fdb6d95d6d75" Nov 03 20:00:20.134980 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.134945 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc"] Nov 03 20:00:20.159386 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.159351 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-10572d-29895600-bsbwc"] Nov 03 20:00:20.187525 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.168662 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl"] Nov 03 20:00:20.209777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.209739 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-225668-29895600-wzxcl"] Nov 03 20:00:20.227664 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.225099 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj"] Nov 03 20:00:20.240912 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.239612 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb"] Nov 03 20:00:20.241791 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.241755 4314 scope.go:122] "RemoveContainer" containerID="3cc7df3bb130ac3348b9ed84a56ccfd060f639b5644328d3efcef5f490797c03" Nov 03 20:00:20.255895 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.255818 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-11c1a8-29895600-h6pcb"] Nov 03 20:00:20.317793 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.314948 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn"] Nov 03 20:00:20.326858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.326827 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fcddeb-29895600-75kwn"] Nov 03 20:00:20.342872 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.342820 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:20.435283 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:20.435254 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985155e7_8d0e_4fe9_96a3_5a59230ac9e5.slice/crio-0ede2852d2c8a6c1d4f785148087dcaf2e29ad1c819f3ee1e82103dacbe63dd9 WatchSource:0}: Error finding container 0ede2852d2c8a6c1d4f785148087dcaf2e29ad1c819f3ee1e82103dacbe63dd9: Status 404 returned error can't find the container with id 0ede2852d2c8a6c1d4f785148087dcaf2e29ad1c819f3ee1e82103dacbe63dd9 Nov 03 20:00:20.442704 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.442638 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0115fa-f263-4cbe-94f6-4671979a6a6f" path="/var/lib/kubelet/pods/4a0115fa-f263-4cbe-94f6-4671979a6a6f/volumes" Nov 03 20:00:20.443325 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.443304 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3" path="/var/lib/kubelet/pods/5cae6a2c-09ce-4228-bd9c-f6bb42e5b6b3/volumes" Nov 03 20:00:20.443937 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.443919 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae" path="/var/lib/kubelet/pods/5e6cdd3a-7bfe-4c46-8d99-4ff711fa8fae/volumes" Nov 03 20:00:20.444952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.444930 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c332276-3c4a-4e62-a917-f7070cb1ceff" path="/var/lib/kubelet/pods/7c332276-3c4a-4e62-a917-f7070cb1ceff/volumes" Nov 03 20:00:20.445558 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.445538 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee99a2b-22ad-49b8-bdd6-939337636258" path="/var/lib/kubelet/pods/7ee99a2b-22ad-49b8-bdd6-939337636258/volumes" Nov 03 20:00:20.448024 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.448007 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz"] Nov 03 20:00:20.508936 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.508908 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573cb6f9-f30c-4f4f-b277-58be9487f1d3-kube-api-access-bbbvq\" (UniqueName: \"kubernetes.io/projected/573cb6f9-f30c-4f4f-b277-58be9487f1d3-kube-api-access-bbbvq\") pod \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " Nov 03 20:00:20.509375 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.509242 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573cb6f9-f30c-4f4f-b277-58be9487f1d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cb6f9-f30c-4f4f-b277-58be9487f1d3-maas-api-service-ca\") pod \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\" (UID: \"573cb6f9-f30c-4f4f-b277-58be9487f1d3\") " Nov 03 20:00:20.509674 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.509648 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573cb6f9-f30c-4f4f-b277-58be9487f1d3-maas-api-service-ca" pod "573cb6f9-f30c-4f4f-b277-58be9487f1d3" (UID: "573cb6f9-f30c-4f4f-b277-58be9487f1d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:20.510636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.510609 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/573cb6f9-f30c-4f4f-b277-58be9487f1d3-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:20.512221 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.512201 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573cb6f9-f30c-4f4f-b277-58be9487f1d3-kube-api-access-bbbvq" pod "573cb6f9-f30c-4f4f-b277-58be9487f1d3" (UID: "573cb6f9-f30c-4f4f-b277-58be9487f1d3"). InnerVolumeSpecName "kube-api-access-bbbvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:20.524688 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:00:20.524661 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cccdd6a_ab8b_4957_a09e_5c0add6e6418.slice/crio-4fe883689c50d344791a60c6062cc49b937d854b77d525b64f47c5eafd968793 WatchSource:0}: Error finding container 4fe883689c50d344791a60c6062cc49b937d854b77d525b64f47c5eafd968793: Status 404 returned error can't find the container with id 4fe883689c50d344791a60c6062cc49b937d854b77d525b64f47c5eafd968793 Nov 03 20:00:20.525501 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.525478 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq"] Nov 03 20:00:20.612448 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.612362 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbbvq\" (UniqueName: \"kubernetes.io/projected/573cb6f9-f30c-4f4f-b277-58be9487f1d3-kube-api-access-bbbvq\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:20.968336 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.968285 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" event={"ID":"0cccdd6a-ab8b-4957-a09e-5c0add6e6418","Type":"ContainerStarted","Data":"4fe883689c50d344791a60c6062cc49b937d854b77d525b64f47c5eafd968793"} Nov 03 20:00:20.971570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.971540 4314 generic.go:356] "Generic (PLEG): container finished" podID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerID="1905cbe4e59a37eb45dcf528c3ea7404bcd16e5c7bfa1619eb93b8d3d4be5287" exitCode=60 Nov 03 20:00:20.971732 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.971708 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerDied","Data":"1905cbe4e59a37eb45dcf528c3ea7404bcd16e5c7bfa1619eb93b8d3d4be5287"} Nov 03 20:00:20.972040 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.972021 4314 scope.go:122] "RemoveContainer" containerID="1905cbe4e59a37eb45dcf528c3ea7404bcd16e5c7bfa1619eb93b8d3d4be5287" Nov 03 20:00:20.973827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.973738 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerStarted","Data":"d7b58be06bcc9ac97470dccef199bc635ac0d9fbb11aa7ba9f1780ab328864ff"} Nov 03 20:00:20.973827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.973794 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerStarted","Data":"e462ecbf5c629d48a06cc8719dc881c36abbab8c67e74755e9f5cd168e33c26a"} Nov 03 20:00:20.979342 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.979313 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" event={"ID":"985155e7-8d0e-4fe9-96a3-5a59230ac9e5","Type":"ContainerStarted","Data":"0ede2852d2c8a6c1d4f785148087dcaf2e29ad1c819f3ee1e82103dacbe63dd9"} Nov 03 20:00:20.981175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.981108 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" Nov 03 20:00:20.981175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.981113 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6" event={"ID":"573cb6f9-f30c-4f4f-b277-58be9487f1d3","Type":"ContainerDied","Data":"e578c0aaa406913e34fedc685d52f53fec8181c22fdbc1217d9f38eb08b82f1c"} Nov 03 20:00:20.981175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.981168 4314 scope.go:122] "RemoveContainer" containerID="3590bdba38c1cc1d394858f8be0d3546d8c5218bd6fd3230e0c5843b092b8639" Nov 03 20:00:20.997693 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:20.997662 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" event={"ID":"d2691ff8-da25-4a4f-aec1-f748b2d097fa","Type":"ContainerStarted","Data":"f015db28dfbbed8bcc7e2af8094dbb51daa1298295c649794f895530681f8cd0"} Nov 03 20:00:21.010384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:21.010343 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" podStartSLOduration=21.010329789 podStartE2EDuration="21.010329789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:21.007744656 +0000 UTC m=+1728.712583997" watchObservedRunningTime="2026-11-03 20:00:21.010329789 +0000 UTC m=+1728.715169129" Nov 03 20:00:21.033204 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:21.033160 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" podStartSLOduration=21.033147506 podStartE2EDuration="21.033147506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:21.028608061 +0000 UTC m=+1728.733447405" watchObservedRunningTime="2026-11-03 20:00:21.033147506 +0000 UTC m=+1728.737986845" Nov 03 20:00:21.052077 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:21.052046 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6"] Nov 03 20:00:21.056617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:21.056568 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895600-ch2t6"] Nov 03 20:00:21.443044 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:21.442976 4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405b9d8b_90d1_4e78_aa4b_efc2dd910cff.slice/crio-conmon-d7b58be06bcc9ac97470dccef199bc635ac0d9fbb11aa7ba9f1780ab328864ff.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:00:21.991178 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:21.991150 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:00:22.009236 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.009205 4314 generic.go:356] "Generic (PLEG): container finished" podID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerID="e5aaf0ba4c2f9968487fca60eb75114d6c2414bcae8bffde85cff161ffa2f875" exitCode=60 Nov 03 20:00:22.009403 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.009267 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" event={"ID":"d2691ff8-da25-4a4f-aec1-f748b2d097fa","Type":"ContainerDied","Data":"e5aaf0ba4c2f9968487fca60eb75114d6c2414bcae8bffde85cff161ffa2f875"} Nov 03 20:00:22.009721 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.009704 4314 scope.go:122] "RemoveContainer" containerID="e5aaf0ba4c2f9968487fca60eb75114d6c2414bcae8bffde85cff161ffa2f875" Nov 03 20:00:22.011149 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.011129 4314 generic.go:356] "Generic (PLEG): container finished" podID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerID="d6a55024d83a925c6d6e8e283967d14b9702e8929a6f0f44885184002f49f19e" exitCode=60 Nov 03 20:00:22.011271 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.011249 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" event={"ID":"0cccdd6a-ab8b-4957-a09e-5c0add6e6418","Type":"ContainerDied","Data":"d6a55024d83a925c6d6e8e283967d14b9702e8929a6f0f44885184002f49f19e"} Nov 03 20:00:22.011495 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.011479 4314 scope.go:122] "RemoveContainer" containerID="d6a55024d83a925c6d6e8e283967d14b9702e8929a6f0f44885184002f49f19e" Nov 03 20:00:22.015622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.015551 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerStarted","Data":"a9084af44643a341f6004492c30d85c3176a284eab4e2ed7e8520d337ff81131"} Nov 03 20:00:22.017216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.017194 4314 generic.go:356] "Generic (PLEG): container finished" podID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerID="d7b58be06bcc9ac97470dccef199bc635ac0d9fbb11aa7ba9f1780ab328864ff" exitCode=7 Nov 03 20:00:22.017422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.017401 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerDied","Data":"d7b58be06bcc9ac97470dccef199bc635ac0d9fbb11aa7ba9f1780ab328864ff"} Nov 03 20:00:22.017778 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.017750 4314 scope.go:122] "RemoveContainer" containerID="d7b58be06bcc9ac97470dccef199bc635ac0d9fbb11aa7ba9f1780ab328864ff" Nov 03 20:00:22.019281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.019263 4314 generic.go:356] "Generic (PLEG): container finished" podID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerID="0dbadcd3fb5026ee990efd8ced757a8ddfcbb665e8482078c1e9d3233ea6a079" exitCode=60 Nov 03 20:00:22.019363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.019310 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" event={"ID":"985155e7-8d0e-4fe9-96a3-5a59230ac9e5","Type":"ContainerDied","Data":"0dbadcd3fb5026ee990efd8ced757a8ddfcbb665e8482078c1e9d3233ea6a079"} Nov 03 20:00:22.019558 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.019540 4314 scope.go:122] "RemoveContainer" containerID="0dbadcd3fb5026ee990efd8ced757a8ddfcbb665e8482078c1e9d3233ea6a079" Nov 03 20:00:22.461377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:22.461332 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" path="/var/lib/kubelet/pods/573cb6f9-f30c-4f4f-b277-58be9487f1d3/volumes" Nov 03 20:00:23.035342 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.034219 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" event={"ID":"0cccdd6a-ab8b-4957-a09e-5c0add6e6418","Type":"ContainerStarted","Data":"b7cf7039cb31d7d9265dec57ec8016b7affd7e8f2afd4602cbcb773be2b6090d"} Nov 03 20:00:23.040425 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.040393 4314 generic.go:356] "Generic (PLEG): container finished" podID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerID="a9084af44643a341f6004492c30d85c3176a284eab4e2ed7e8520d337ff81131" exitCode=60 Nov 03 20:00:23.040615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.040479 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerDied","Data":"a9084af44643a341f6004492c30d85c3176a284eab4e2ed7e8520d337ff81131"} Nov 03 20:00:23.040615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.040525 4314 scope.go:122] "RemoveContainer" containerID="1905cbe4e59a37eb45dcf528c3ea7404bcd16e5c7bfa1619eb93b8d3d4be5287" Nov 03 20:00:23.040867 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.040841 4314 scope.go:122] "RemoveContainer" containerID="a9084af44643a341f6004492c30d85c3176a284eab4e2ed7e8520d337ff81131" Nov 03 20:00:23.041273 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:23.041241 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j_odh-ai-gateway-infra(4288e8ce-c0ad-47c6-bb22-4648dfc617ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" Nov 03 20:00:23.043038 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.043010 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerStarted","Data":"585f6e5baa3e2d3c14482893548c855fbf3df38347f8671087cf1755ca9037df"} Nov 03 20:00:23.045470 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.045445 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" event={"ID":"985155e7-8d0e-4fe9-96a3-5a59230ac9e5","Type":"ContainerStarted","Data":"28141520c5ad7a6692d5d9b5bd6e53b5250c29bf7befc560d50f112fea683403"} Nov 03 20:00:23.047936 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.047912 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" event={"ID":"d2691ff8-da25-4a4f-aec1-f748b2d097fa","Type":"ContainerStarted","Data":"858a88e0927d9f31bb7dc26fb4cabef5c00149b9f44768861ea9da1e90566fd5"} Nov 03 20:00:23.052284 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.052245 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" podStartSLOduration=23.052234873 podStartE2EDuration="23.052234873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:23.051167374 +0000 UTC m=+1730.756006715" watchObservedRunningTime="2026-11-03 20:00:23.052234873 +0000 UTC m=+1730.757074217" Nov 03 20:00:23.079300 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:23.079256 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" podStartSLOduration=23.079242206 podStartE2EDuration="23.079242206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:23.072710338 +0000 UTC m=+1730.777549679" watchObservedRunningTime="2026-11-03 20:00:23.079242206 +0000 UTC m=+1730.784081545" Nov 03 20:00:24.061329 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.061296 4314 generic.go:356] "Generic (PLEG): container finished" podID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerID="28141520c5ad7a6692d5d9b5bd6e53b5250c29bf7befc560d50f112fea683403" exitCode=60 Nov 03 20:00:24.061880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.061398 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" event={"ID":"985155e7-8d0e-4fe9-96a3-5a59230ac9e5","Type":"ContainerDied","Data":"28141520c5ad7a6692d5d9b5bd6e53b5250c29bf7befc560d50f112fea683403"} Nov 03 20:00:24.061880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.061491 4314 scope.go:122] "RemoveContainer" containerID="0dbadcd3fb5026ee990efd8ced757a8ddfcbb665e8482078c1e9d3233ea6a079" Nov 03 20:00:24.073223 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.062524 4314 scope.go:122] "RemoveContainer" containerID="28141520c5ad7a6692d5d9b5bd6e53b5250c29bf7befc560d50f112fea683403" Nov 03 20:00:24.073223 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:24.063709 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz_odh-ai-gateway-infra(985155e7-8d0e-4fe9-96a3-5a59230ac9e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" Nov 03 20:00:24.073223 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.064997 4314 generic.go:356] "Generic (PLEG): container finished" podID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerID="858a88e0927d9f31bb7dc26fb4cabef5c00149b9f44768861ea9da1e90566fd5" exitCode=60 Nov 03 20:00:24.073223 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.065169 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" event={"ID":"d2691ff8-da25-4a4f-aec1-f748b2d097fa","Type":"ContainerDied","Data":"858a88e0927d9f31bb7dc26fb4cabef5c00149b9f44768861ea9da1e90566fd5"} Nov 03 20:00:24.073223 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.065554 4314 scope.go:122] "RemoveContainer" containerID="858a88e0927d9f31bb7dc26fb4cabef5c00149b9f44768861ea9da1e90566fd5" Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:24.074912 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj_odh-ai-gateway-infra(d2691ff8-da25-4a4f-aec1-f748b2d097fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.082459 4314 generic.go:356] "Generic (PLEG): container finished" podID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerID="b7cf7039cb31d7d9265dec57ec8016b7affd7e8f2afd4602cbcb773be2b6090d" exitCode=60 Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.082626 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" event={"ID":"0cccdd6a-ab8b-4957-a09e-5c0add6e6418","Type":"ContainerDied","Data":"b7cf7039cb31d7d9265dec57ec8016b7affd7e8f2afd4602cbcb773be2b6090d"} Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.083055 4314 scope.go:122] "RemoveContainer" containerID="b7cf7039cb31d7d9265dec57ec8016b7affd7e8f2afd4602cbcb773be2b6090d" Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:24.083455 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq_odh-ai-gateway-infra(0cccdd6a-ab8b-4957-a09e-5c0add6e6418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.086138 4314 scope.go:122] "RemoveContainer" containerID="a9084af44643a341f6004492c30d85c3176a284eab4e2ed7e8520d337ff81131" Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:24.086486 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j_odh-ai-gateway-infra(4288e8ce-c0ad-47c6-bb22-4648dfc617ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.090719 4314 generic.go:356] "Generic (PLEG): container finished" podID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerID="585f6e5baa3e2d3c14482893548c855fbf3df38347f8671087cf1755ca9037df" exitCode=7 Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.090787 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerDied","Data":"585f6e5baa3e2d3c14482893548c855fbf3df38347f8671087cf1755ca9037df"} Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.090979 4314 scope.go:122] "RemoveContainer" containerID="585f6e5baa3e2d3c14482893548c855fbf3df38347f8671087cf1755ca9037df" Nov 03 20:00:24.093824 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:24.091228 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg_odh-ai-gateway-infra(405b9d8b-90d1-4e78-aa4b-efc2dd910cff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" Nov 03 20:00:24.099168 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.099147 4314 scope.go:122] "RemoveContainer" containerID="e5aaf0ba4c2f9968487fca60eb75114d6c2414bcae8bffde85cff161ffa2f875" Nov 03 20:00:24.117150 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.117132 4314 scope.go:122] "RemoveContainer" containerID="d6a55024d83a925c6d6e8e283967d14b9702e8929a6f0f44885184002f49f19e" Nov 03 20:00:24.132563 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:24.132535 4314 scope.go:122] "RemoveContainer" containerID="d7b58be06bcc9ac97470dccef199bc635ac0d9fbb11aa7ba9f1780ab328864ff" Nov 03 20:00:25.105223 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:25.105189 4314 scope.go:122] "RemoveContainer" containerID="858a88e0927d9f31bb7dc26fb4cabef5c00149b9f44768861ea9da1e90566fd5" Nov 03 20:00:25.105716 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:25.105686 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj_odh-ai-gateway-infra(d2691ff8-da25-4a4f-aec1-f748b2d097fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" Nov 03 20:00:25.107098 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:25.107070 4314 scope.go:122] "RemoveContainer" containerID="b7cf7039cb31d7d9265dec57ec8016b7affd7e8f2afd4602cbcb773be2b6090d" Nov 03 20:00:25.107468 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:25.107445 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq_odh-ai-gateway-infra(0cccdd6a-ab8b-4957-a09e-5c0add6e6418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" Nov 03 20:00:25.108838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:25.108817 4314 scope.go:122] "RemoveContainer" containerID="585f6e5baa3e2d3c14482893548c855fbf3df38347f8671087cf1755ca9037df" Nov 03 20:00:25.109129 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:25.109109 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg_odh-ai-gateway-infra(405b9d8b-90d1-4e78-aa4b-efc2dd910cff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" Nov 03 20:00:25.110533 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:25.110514 4314 scope.go:122] "RemoveContainer" containerID="28141520c5ad7a6692d5d9b5bd6e53b5250c29bf7befc560d50f112fea683403" Nov 03 20:00:25.110849 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:25.110825 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz_odh-ai-gateway-infra(985155e7-8d0e-4fe9-96a3-5a59230ac9e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" Nov 03 20:00:34.421303 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:34.421268 4314 scope.go:122] "RemoveContainer" containerID="585f6e5baa3e2d3c14482893548c855fbf3df38347f8671087cf1755ca9037df" Nov 03 20:00:35.295226 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:35.295191 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerStarted","Data":"3fa2e4a5dfb1b809dcd4a0b9cf53b9df8e9b00a54f5c5e88a76a2f7b3c495c82"} Nov 03 20:00:35.420960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:35.420919 4314 scope.go:122] "RemoveContainer" containerID="28141520c5ad7a6692d5d9b5bd6e53b5250c29bf7befc560d50f112fea683403" Nov 03 20:00:35.422015 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:35.421880 4314 scope.go:122] "RemoveContainer" containerID="a9084af44643a341f6004492c30d85c3176a284eab4e2ed7e8520d337ff81131" Nov 03 20:00:35.422834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:35.422453 4314 scope.go:122] "RemoveContainer" containerID="b7cf7039cb31d7d9265dec57ec8016b7affd7e8f2afd4602cbcb773be2b6090d" Nov 03 20:00:35.423673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:35.423127 4314 scope.go:122] "RemoveContainer" containerID="858a88e0927d9f31bb7dc26fb4cabef5c00149b9f44768861ea9da1e90566fd5" Nov 03 20:00:36.310077 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.310043 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" event={"ID":"985155e7-8d0e-4fe9-96a3-5a59230ac9e5","Type":"ContainerStarted","Data":"61688138504e7992f8275dfb2168b3f4cfc1fa9286e0ea37f4909dbdde7655d7"} Nov 03 20:00:36.312037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.312007 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" event={"ID":"d2691ff8-da25-4a4f-aec1-f748b2d097fa","Type":"ContainerStarted","Data":"a4aad2c99d248558728a733b25be951c4aa5e5b00699419729f86761f2deca58"} Nov 03 20:00:36.313949 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.313920 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" event={"ID":"0cccdd6a-ab8b-4957-a09e-5c0add6e6418","Type":"ContainerStarted","Data":"d5e25d32bff0736ff6aa16fc56d3cb55be455bb0b62feed77c7a6442a88b6347"} Nov 03 20:00:36.315802 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.315775 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerStarted","Data":"9ff95b25e5c8ed152d4b0df0e50d85a9c2c5b8a4ab865ff892a398db5f431a37"} Nov 03 20:00:36.317664 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.317639 4314 generic.go:356] "Generic (PLEG): container finished" podID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerID="3fa2e4a5dfb1b809dcd4a0b9cf53b9df8e9b00a54f5c5e88a76a2f7b3c495c82" exitCode=7 Nov 03 20:00:36.317752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.317703 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerDied","Data":"3fa2e4a5dfb1b809dcd4a0b9cf53b9df8e9b00a54f5c5e88a76a2f7b3c495c82"} Nov 03 20:00:36.317820 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.317777 4314 scope.go:122] "RemoveContainer" containerID="585f6e5baa3e2d3c14482893548c855fbf3df38347f8671087cf1755ca9037df" Nov 03 20:00:36.318085 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.318064 4314 scope.go:122] "RemoveContainer" containerID="3fa2e4a5dfb1b809dcd4a0b9cf53b9df8e9b00a54f5c5e88a76a2f7b3c495c82" Nov 03 20:00:36.318444 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:36.318417 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg_odh-ai-gateway-infra(405b9d8b-90d1-4e78-aa4b-efc2dd910cff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" Nov 03 20:00:36.324169 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:36.324143 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg"] Nov 03 20:00:37.328512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.328476 4314 generic.go:356] "Generic (PLEG): container finished" podID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerID="d5e25d32bff0736ff6aa16fc56d3cb55be455bb0b62feed77c7a6442a88b6347" exitCode=60 Nov 03 20:00:37.329003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.328540 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" event={"ID":"0cccdd6a-ab8b-4957-a09e-5c0add6e6418","Type":"ContainerDied","Data":"d5e25d32bff0736ff6aa16fc56d3cb55be455bb0b62feed77c7a6442a88b6347"} Nov 03 20:00:37.329003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.328633 4314 scope.go:122] "RemoveContainer" containerID="b7cf7039cb31d7d9265dec57ec8016b7affd7e8f2afd4602cbcb773be2b6090d" Nov 03 20:00:37.329413 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.329390 4314 scope.go:122] "RemoveContainer" containerID="d5e25d32bff0736ff6aa16fc56d3cb55be455bb0b62feed77c7a6442a88b6347" Nov 03 20:00:37.329955 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:37.329924 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq_odh-ai-gateway-infra(0cccdd6a-ab8b-4957-a09e-5c0add6e6418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" Nov 03 20:00:37.330782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.330758 4314 generic.go:356] "Generic (PLEG): container finished" podID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerID="9ff95b25e5c8ed152d4b0df0e50d85a9c2c5b8a4ab865ff892a398db5f431a37" exitCode=60 Nov 03 20:00:37.330865 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.330826 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerDied","Data":"9ff95b25e5c8ed152d4b0df0e50d85a9c2c5b8a4ab865ff892a398db5f431a37"} Nov 03 20:00:37.331276 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.331256 4314 scope.go:122] "RemoveContainer" containerID="9ff95b25e5c8ed152d4b0df0e50d85a9c2c5b8a4ab865ff892a398db5f431a37" Nov 03 20:00:37.331726 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:37.331703 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j_odh-ai-gateway-infra(4288e8ce-c0ad-47c6-bb22-4648dfc617ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" Nov 03 20:00:37.335151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.335120 4314 generic.go:356] "Generic (PLEG): container finished" podID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerID="61688138504e7992f8275dfb2168b3f4cfc1fa9286e0ea37f4909dbdde7655d7" exitCode=60 Nov 03 20:00:37.335514 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.335239 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" event={"ID":"985155e7-8d0e-4fe9-96a3-5a59230ac9e5","Type":"ContainerDied","Data":"61688138504e7992f8275dfb2168b3f4cfc1fa9286e0ea37f4909dbdde7655d7"} Nov 03 20:00:37.335788 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.335763 4314 scope.go:122] "RemoveContainer" containerID="61688138504e7992f8275dfb2168b3f4cfc1fa9286e0ea37f4909dbdde7655d7" Nov 03 20:00:37.336312 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:37.336274 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz_odh-ai-gateway-infra(985155e7-8d0e-4fe9-96a3-5a59230ac9e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" Nov 03 20:00:37.337894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.337871 4314 generic.go:356] "Generic (PLEG): container finished" podID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerID="a4aad2c99d248558728a733b25be951c4aa5e5b00699419729f86761f2deca58" exitCode=60 Nov 03 20:00:37.337998 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.337902 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" event={"ID":"d2691ff8-da25-4a4f-aec1-f748b2d097fa","Type":"ContainerDied","Data":"a4aad2c99d248558728a733b25be951c4aa5e5b00699419729f86761f2deca58"} Nov 03 20:00:37.338246 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.338229 4314 scope.go:122] "RemoveContainer" containerID="a4aad2c99d248558728a733b25be951c4aa5e5b00699419729f86761f2deca58" Nov 03 20:00:37.338618 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:37.338575 4314 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj_odh-ai-gateway-infra(d2691ff8-da25-4a4f-aec1-f748b2d097fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" Nov 03 20:00:37.346469 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.346445 4314 scope.go:122] "RemoveContainer" containerID="a9084af44643a341f6004492c30d85c3176a284eab4e2ed7e8520d337ff81131" Nov 03 20:00:37.347607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.347573 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz"] Nov 03 20:00:37.368365 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.368216 4314 scope.go:122] "RemoveContainer" containerID="28141520c5ad7a6692d5d9b5bd6e53b5250c29bf7befc560d50f112fea683403" Nov 03 20:00:37.384307 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.382888 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj"] Nov 03 20:00:37.392985 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.392964 4314 scope.go:122] "RemoveContainer" containerID="858a88e0927d9f31bb7dc26fb4cabef5c00149b9f44768861ea9da1e90566fd5" Nov 03 20:00:37.410483 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.410453 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq"] Nov 03 20:00:37.438731 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.438639 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j"] Nov 03 20:00:37.631650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.631623 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:37.790303 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.788885 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-kube-api-access-dv5cw\" (UniqueName: \"kubernetes.io/projected/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-kube-api-access-dv5cw\") pod \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " Nov 03 20:00:37.790303 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.789011 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-maas-api-service-ca\") pod \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\" (UID: \"405b9d8b-90d1-4e78-aa4b-efc2dd910cff\") " Nov 03 20:00:37.790303 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.789931 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-maas-api-service-ca" pod "405b9d8b-90d1-4e78-aa4b-efc2dd910cff" (UID: "405b9d8b-90d1-4e78-aa4b-efc2dd910cff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:37.790303 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.790163 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:37.791928 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.791896 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-kube-api-access-dv5cw" pod "405b9d8b-90d1-4e78-aa4b-efc2dd910cff" (UID: "405b9d8b-90d1-4e78-aa4b-efc2dd910cff"). InnerVolumeSpecName "kube-api-access-dv5cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:37.891811 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:37.891727 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv5cw\" (UniqueName: \"kubernetes.io/projected/405b9d8b-90d1-4e78-aa4b-efc2dd910cff-kube-api-access-dv5cw\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:38.377709 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.377667 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" event={"ID":"405b9d8b-90d1-4e78-aa4b-efc2dd910cff","Type":"ContainerDied","Data":"e462ecbf5c629d48a06cc8719dc881c36abbab8c67e74755e9f5cd168e33c26a"} Nov 03 20:00:38.378106 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.377772 4314 scope.go:122] "RemoveContainer" containerID="3fa2e4a5dfb1b809dcd4a0b9cf53b9df8e9b00a54f5c5e88a76a2f7b3c495c82" Nov 03 20:00:38.378392 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.378371 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg" Nov 03 20:00:38.463041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.463013 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg"] Nov 03 20:00:38.504800 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.490426 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2747a4-29895600-k4hdg"] Nov 03 20:00:38.686614 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.671081 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:38.807039 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.807010 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:38.827015 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.826977 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2691ff8-da25-4a4f-aec1-f748b2d097fa-kube-api-access-88krx\" (UniqueName: \"kubernetes.io/projected/d2691ff8-da25-4a4f-aec1-f748b2d097fa-kube-api-access-88krx\") pod \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " Nov 03 20:00:38.827261 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.827240 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2691ff8-da25-4a4f-aec1-f748b2d097fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2691ff8-da25-4a4f-aec1-f748b2d097fa-maas-api-service-ca\") pod \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\" (UID: \"d2691ff8-da25-4a4f-aec1-f748b2d097fa\") " Nov 03 20:00:38.833814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.829210 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2691ff8-da25-4a4f-aec1-f748b2d097fa-maas-api-service-ca" pod "d2691ff8-da25-4a4f-aec1-f748b2d097fa" (UID: "d2691ff8-da25-4a4f-aec1-f748b2d097fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:38.843376 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.843350 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2691ff8-da25-4a4f-aec1-f748b2d097fa-kube-api-access-88krx" pod "d2691ff8-da25-4a4f-aec1-f748b2d097fa" (UID: "d2691ff8-da25-4a4f-aec1-f748b2d097fa"). InnerVolumeSpecName "kube-api-access-88krx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:38.934871 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.934535 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-kube-api-access-ngk5k\" (UniqueName: \"kubernetes.io/projected/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-kube-api-access-ngk5k\") pod \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " Nov 03 20:00:38.934871 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.934677 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-maas-api-service-ca\") pod \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\" (UID: \"0cccdd6a-ab8b-4957-a09e-5c0add6e6418\") " Nov 03 20:00:38.949064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.941880 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-maas-api-service-ca" pod "0cccdd6a-ab8b-4957-a09e-5c0add6e6418" (UID: "0cccdd6a-ab8b-4957-a09e-5c0add6e6418"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:38.949064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.944764 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88krx\" (UniqueName: \"kubernetes.io/projected/d2691ff8-da25-4a4f-aec1-f748b2d097fa-kube-api-access-88krx\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:38.949064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.944789 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2691ff8-da25-4a4f-aec1-f748b2d097fa-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:38.949064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.944798 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:38.960756 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.959803 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-kube-api-access-ngk5k" pod "0cccdd6a-ab8b-4957-a09e-5c0add6e6418" (UID: "0cccdd6a-ab8b-4957-a09e-5c0add6e6418"). InnerVolumeSpecName "kube-api-access-ngk5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:38.963299 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.963269 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:38.994633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:38.989593 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:39.047575 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.047543 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-maas-api-service-ca\") pod \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " Nov 03 20:00:39.047781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.047748 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-kube-api-access-plvk9\" (UniqueName: \"kubernetes.io/projected/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-kube-api-access-plvk9\") pod \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " Nov 03 20:00:39.047867 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.047815 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-maas-api-service-ca\") pod \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\" (UID: \"4288e8ce-c0ad-47c6-bb22-4648dfc617ea\") " Nov 03 20:00:39.047942 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.047926 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-kube-api-access-d6fjv\" (UniqueName: \"kubernetes.io/projected/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-kube-api-access-d6fjv\") pod \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\" (UID: \"985155e7-8d0e-4fe9-96a3-5a59230ac9e5\") " Nov 03 20:00:39.048523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.048500 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngk5k\" (UniqueName: \"kubernetes.io/projected/0cccdd6a-ab8b-4957-a09e-5c0add6e6418-kube-api-access-ngk5k\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:39.049813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.049463 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-maas-api-service-ca" pod "985155e7-8d0e-4fe9-96a3-5a59230ac9e5" (UID: "985155e7-8d0e-4fe9-96a3-5a59230ac9e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:39.050875 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.050477 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-maas-api-service-ca" pod "4288e8ce-c0ad-47c6-bb22-4648dfc617ea" (UID: "4288e8ce-c0ad-47c6-bb22-4648dfc617ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:39.052129 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.052107 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-kube-api-access-plvk9" pod "4288e8ce-c0ad-47c6-bb22-4648dfc617ea" (UID: "4288e8ce-c0ad-47c6-bb22-4648dfc617ea"). InnerVolumeSpecName "kube-api-access-plvk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:39.066435 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.066404 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-kube-api-access-d6fjv" pod "985155e7-8d0e-4fe9-96a3-5a59230ac9e5" (UID: "985155e7-8d0e-4fe9-96a3-5a59230ac9e5"). InnerVolumeSpecName "kube-api-access-d6fjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:39.150038 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.150005 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:39.150038 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.150038 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6fjv\" (UniqueName: \"kubernetes.io/projected/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-kube-api-access-d6fjv\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:39.150469 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.150053 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/985155e7-8d0e-4fe9-96a3-5a59230ac9e5-maas-api-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:39.150469 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.150065 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvk9\" (UniqueName: \"kubernetes.io/projected/4288e8ce-c0ad-47c6-bb22-4648dfc617ea-kube-api-access-plvk9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:39.395606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.395443 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" event={"ID":"985155e7-8d0e-4fe9-96a3-5a59230ac9e5","Type":"ContainerDied","Data":"0ede2852d2c8a6c1d4f785148087dcaf2e29ad1c819f3ee1e82103dacbe63dd9"} Nov 03 20:00:39.395606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.395518 4314 scope.go:122] "RemoveContainer" containerID="61688138504e7992f8275dfb2168b3f4cfc1fa9286e0ea37f4909dbdde7655d7" Nov 03 20:00:39.396270 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.395637 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz" Nov 03 20:00:39.410842 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.409146 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" event={"ID":"d2691ff8-da25-4a4f-aec1-f748b2d097fa","Type":"ContainerDied","Data":"f015db28dfbbed8bcc7e2af8094dbb51daa1298295c649794f895530681f8cd0"} Nov 03 20:00:39.410842 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.409255 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj" Nov 03 20:00:39.413314 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.413287 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" Nov 03 20:00:39.413542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.413522 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq" event={"ID":"0cccdd6a-ab8b-4957-a09e-5c0add6e6418","Type":"ContainerDied","Data":"4fe883689c50d344791a60c6062cc49b937d854b77d525b64f47c5eafd968793"} Nov 03 20:00:39.415648 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.415620 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" event={"ID":"4288e8ce-c0ad-47c6-bb22-4648dfc617ea","Type":"ContainerDied","Data":"f04ddc64114ac0c3251afb253e35fbe8e288d539ef3f47d2d7c538b1f5abf6f3"} Nov 03 20:00:39.415752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.415705 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j" Nov 03 20:00:39.424995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.424979 4314 scope.go:122] "RemoveContainer" containerID="a4aad2c99d248558728a733b25be951c4aa5e5b00699419729f86761f2deca58" Nov 03 20:00:39.445668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.445340 4314 scope.go:122] "RemoveContainer" containerID="d5e25d32bff0736ff6aa16fc56d3cb55be455bb0b62feed77c7a6442a88b6347" Nov 03 20:00:39.456102 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.455879 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz"] Nov 03 20:00:39.457984 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.457966 4314 scope.go:122] "RemoveContainer" containerID="9ff95b25e5c8ed152d4b0df0e50d85a9c2c5b8a4ab865ff892a398db5f431a37" Nov 03 20:00:39.464222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.464197 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ffe132-29895600-9x7mz"] Nov 03 20:00:39.482822 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.481620 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq"] Nov 03 20:00:39.490312 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.490080 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7a62d4-29895600-lqtrq"] Nov 03 20:00:39.500480 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.500450 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j"] Nov 03 20:00:39.507817 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.507795 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895600-m2k7j"] Nov 03 20:00:39.517560 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.517537 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj"] Nov 03 20:00:39.521643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:39.521625 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f58171-29895600-mx2dj"] Nov 03 20:00:40.460708 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:40.460658 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" path="/var/lib/kubelet/pods/0cccdd6a-ab8b-4957-a09e-5c0add6e6418/volumes" Nov 03 20:00:40.462159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:40.462137 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" path="/var/lib/kubelet/pods/405b9d8b-90d1-4e78-aa4b-efc2dd910cff/volumes" Nov 03 20:00:40.462872 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:40.462845 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" path="/var/lib/kubelet/pods/4288e8ce-c0ad-47c6-bb22-4648dfc617ea/volumes" Nov 03 20:00:40.464187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:40.464164 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" path="/var/lib/kubelet/pods/985155e7-8d0e-4fe9-96a3-5a59230ac9e5/volumes" Nov 03 20:00:40.464826 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:40.464807 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" path="/var/lib/kubelet/pods/d2691ff8-da25-4a4f-aec1-f748b2d097fa/volumes" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.443360 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4"] Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444711 4314 cpu_manager.go:414] "containerMap: removing container" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444736 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444758 4314 cpu_manager.go:414] "containerMap: removing container" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444766 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444776 4314 cpu_manager.go:414] "containerMap: removing container" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444781 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444791 4314 cpu_manager.go:414] "containerMap: removing container" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444797 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444805 4314 cpu_manager.go:414] "containerMap: removing container" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444813 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444832 4314 cpu_manager.go:414] "containerMap: removing container" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444840 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444853 4314 cpu_manager.go:414] "containerMap: removing container" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444860 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444871 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444875 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444890 4314 cpu_manager.go:414] "containerMap: removing container" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444894 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444903 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444907 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444942 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444947 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444955 4314 cpu_manager.go:414] "containerMap: removing container" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444959 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444965 4314 cpu_manager.go:414] "containerMap: removing container" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444969 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444977 4314 cpu_manager.go:414] "containerMap: removing container" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.444981 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445135 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445143 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445150 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445156 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445164 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445175 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445188 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445197 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445207 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445213 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445221 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445231 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="573cb6f9-f30c-4f4f-b277-58be9487f1d3" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445241 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445254 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:48.445317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.445262 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4288e8ce-c0ad-47c6-bb22-4648dfc617ea" containerName="cleanup" Nov 03 20:00:48.451523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.451501 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4"] Nov 03 20:00:48.451659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.451629 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.460840 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.457631 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:48.479448 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.478021 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:00:48.605363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.605322 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.605546 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.605522 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.605644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.605552 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh9zz\" (UniqueName: \"kubernetes.io/projected/fea4de96-2c62-4fa9-befc-a03e630b88fd-kube-api-access-zh9zz\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.707477 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.707389 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.707687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.707575 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.707687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.707620 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh9zz\" (UniqueName: \"kubernetes.io/projected/fea4de96-2c62-4fa9-befc-a03e630b88fd-kube-api-access-zh9zz\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.708210 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.708188 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.709880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.709857 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.715824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.715803 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh9zz\" (UniqueName: \"kubernetes.io/projected/fea4de96-2c62-4fa9-befc-a03e630b88fd-kube-api-access-zh9zz\") pod \"maas-api-revoke-sub-742f6235-djzb4\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:48.829757 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:48.829729 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:49.143603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:49.139942 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4"] Nov 03 20:00:49.570057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:49.570014 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" event={"ID":"fea4de96-2c62-4fa9-befc-a03e630b88fd","Type":"ContainerStarted","Data":"6afd7d333758b018273276a90b687688f3b57205a6c62446953b84ccd1ce98f9"} Nov 03 20:00:50.580736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:50.580699 4314 generic.go:356] "Generic (PLEG): container finished" podID="fea4de96-2c62-4fa9-befc-a03e630b88fd" containerID="e1fd576617cff36784c3c79716732a31a50f448d2f80f77e5f7f726048a91e39" exitCode=0 Nov 03 20:00:50.581237 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:50.580862 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" event={"ID":"fea4de96-2c62-4fa9-befc-a03e630b88fd","Type":"ContainerDied","Data":"e1fd576617cff36784c3c79716732a31a50f448d2f80f77e5f7f726048a91e39"} Nov 03 20:00:51.896978 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:51.896949 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:51.999662 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:00:51.999625 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:00:52.022483 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.022452 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea4de96-2c62-4fa9-befc-a03e630b88fd-kube-api-access-zh9zz\" (UniqueName: \"kubernetes.io/projected/fea4de96-2c62-4fa9-befc-a03e630b88fd-kube-api-access-zh9zz\") pod \"fea4de96-2c62-4fa9-befc-a03e630b88fd\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " Nov 03 20:00:52.022636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.022510 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-openshift-ca\") pod \"fea4de96-2c62-4fa9-befc-a03e630b88fd\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " Nov 03 20:00:52.022636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.022537 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-xks-ca\") pod \"fea4de96-2c62-4fa9-befc-a03e630b88fd\" (UID: \"fea4de96-2c62-4fa9-befc-a03e630b88fd\") " Nov 03 20:00:52.022930 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.022896 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-openshift-ca" pod "fea4de96-2c62-4fa9-befc-a03e630b88fd" (UID: "fea4de96-2c62-4fa9-befc-a03e630b88fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:52.023402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.023379 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:52.024626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.024604 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea4de96-2c62-4fa9-befc-a03e630b88fd-kube-api-access-zh9zz" pod "fea4de96-2c62-4fa9-befc-a03e630b88fd" (UID: "fea4de96-2c62-4fa9-befc-a03e630b88fd"). InnerVolumeSpecName "kube-api-access-zh9zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:52.024764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.024744 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-xks-ca" pod "fea4de96-2c62-4fa9-befc-a03e630b88fd" (UID: "fea4de96-2c62-4fa9-befc-a03e630b88fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:52.124868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.124784 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh9zz\" (UniqueName: \"kubernetes.io/projected/fea4de96-2c62-4fa9-befc-a03e630b88fd-kube-api-access-zh9zz\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:52.124868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.124824 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea4de96-2c62-4fa9-befc-a03e630b88fd-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:52.609043 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.609007 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" event={"ID":"fea4de96-2c62-4fa9-befc-a03e630b88fd","Type":"ContainerDied","Data":"6afd7d333758b018273276a90b687688f3b57205a6c62446953b84ccd1ce98f9"} Nov 03 20:00:52.609203 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.609060 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6afd7d333758b018273276a90b687688f3b57205a6c62446953b84ccd1ce98f9" Nov 03 20:00:52.609203 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:52.609129 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.328837 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm"] Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330243 4314 cpu_manager.go:414] "containerMap: removing container" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330270 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330315 4314 cpu_manager.go:414] "containerMap: removing container" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330323 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2691ff8-da25-4a4f-aec1-f748b2d097fa" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330337 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fea4de96-2c62-4fa9-befc-a03e630b88fd" containerName="revoke-keys" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330346 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea4de96-2c62-4fa9-befc-a03e630b88fd" containerName="revoke-keys" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330394 4314 cpu_manager.go:414] "containerMap: removing container" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330406 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330422 4314 cpu_manager.go:414] "containerMap: removing container" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330433 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330737 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="405b9d8b-90d1-4e78-aa4b-efc2dd910cff" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330758 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="985155e7-8d0e-4fe9-96a3-5a59230ac9e5" containerName="cleanup" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330778 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea4de96-2c62-4fa9-befc-a03e630b88fd" containerName="revoke-keys" Nov 03 20:00:59.331606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.330798 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cccdd6a-ab8b-4957-a09e-5c0add6e6418" containerName="cleanup" Nov 03 20:00:59.350782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.345657 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.370626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.367918 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:00:59.370626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.368054 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:00:59.370626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.369256 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm"] Nov 03 20:00:59.468602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.467933 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.468602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.468250 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkzr\" (UniqueName: \"kubernetes.io/projected/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-kube-api-access-slkzr\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.468602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.468453 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.570393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.570354 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkzr\" (UniqueName: \"kubernetes.io/projected/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-kube-api-access-slkzr\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.570619 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.570512 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.570726 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.570628 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.571293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.571269 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.572991 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.572964 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.578840 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.578814 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkzr\" (UniqueName: \"kubernetes.io/projected/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-kube-api-access-slkzr\") pod \"maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:00:59.733391 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:00:59.733302 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:01:00.045430 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:01:00.045399 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1e9cb4_5a73_43ce_bac0_a56b36bb3f8a.slice/crio-9be69e3fab784cb19f6d799af4147129ef5f062b65063735d1f24d4009de304d WatchSource:0}: Error finding container 9be69e3fab784cb19f6d799af4147129ef5f062b65063735d1f24d4009de304d: Status 404 returned error can't find the container with id 9be69e3fab784cb19f6d799af4147129ef5f062b65063735d1f24d4009de304d Nov 03 20:01:00.045570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:00.045480 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm"] Nov 03 20:01:00.691603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:00.689906 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" event={"ID":"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a","Type":"ContainerStarted","Data":"dfaf4842637b31127ca825859a0275acc2f368270b343ff7fac9850c06e1cc05"} Nov 03 20:01:00.691603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:00.689972 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" event={"ID":"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a","Type":"ContainerStarted","Data":"9be69e3fab784cb19f6d799af4147129ef5f062b65063735d1f24d4009de304d"} Nov 03 20:01:00.711929 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:00.711878 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" podStartSLOduration=1.711863701 podStartE2EDuration="1.711863701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:00.70668569 +0000 UTC m=+1768.411525033" watchObservedRunningTime="2026-11-03 20:01:00.711863701 +0000 UTC m=+1768.416703039" Nov 03 20:01:01.703781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:01.703743 4314 generic.go:356] "Generic (PLEG): container finished" podID="1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" containerID="dfaf4842637b31127ca825859a0275acc2f368270b343ff7fac9850c06e1cc05" exitCode=0 Nov 03 20:01:01.704265 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:01.703826 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" event={"ID":"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a","Type":"ContainerDied","Data":"dfaf4842637b31127ca825859a0275acc2f368270b343ff7fac9850c06e1cc05"} Nov 03 20:01:02.976454 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:02.976385 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:01:03.079678 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.079633 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-openshift-ca\") pod \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " Nov 03 20:01:03.079912 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.079894 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-xks-ca\") pod \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " Nov 03 20:01:03.080003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.079964 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-kube-api-access-slkzr\" (UniqueName: \"kubernetes.io/projected/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-kube-api-access-slkzr\") pod \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\" (UID: \"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a\") " Nov 03 20:01:03.080003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.079971 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-openshift-ca" pod "1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" (UID: "1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:03.080951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.080925 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:03.082767 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.082735 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-kube-api-access-slkzr" pod "1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" (UID: "1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a"). InnerVolumeSpecName "kube-api-access-slkzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:03.082886 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.082835 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-xks-ca" pod "1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" (UID: "1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:03.183147 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.183116 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:03.183147 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.183144 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkzr\" (UniqueName: \"kubernetes.io/projected/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a-kube-api-access-slkzr\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:03.734153 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.734124 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" Nov 03 20:01:03.734354 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.734161 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm" event={"ID":"1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a","Type":"ContainerDied","Data":"9be69e3fab784cb19f6d799af4147129ef5f062b65063735d1f24d4009de304d"} Nov 03 20:01:03.734354 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:03.734221 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be69e3fab784cb19f6d799af4147129ef5f062b65063735d1f24d4009de304d" Nov 03 20:01:16.230626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:16.225122 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm"] Nov 03 20:01:16.236639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:16.236303 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bdb694-74gtm"] Nov 03 20:01:16.437358 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:16.437325 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" path="/var/lib/kubelet/pods/1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a/volumes" Nov 03 20:01:22.011018 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:01:22.010971 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:01:23.122062 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:23.122026 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv"] Nov 03 20:01:23.122880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:23.122827 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" containerID="cri-o://4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953" gracePeriod=30 Nov 03 20:01:28.446041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.446011 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 20:01:28.491636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.491604 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv"] Nov 03 20:01:28.493157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.493129 4314 cpu_manager.go:414] "containerMap: removing container" podUID="1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" containerName="revoke-keys" Nov 03 20:01:28.493314 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.493299 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" containerName="revoke-keys" Nov 03 20:01:28.493520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.493504 4314 cpu_manager.go:414] "containerMap: removing container" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" Nov 03 20:01:28.493646 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.493629 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" Nov 03 20:01:28.493789 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.493776 4314 cpu_manager.go:414] "containerMap: removing container" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" Nov 03 20:01:28.493890 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.493876 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" Nov 03 20:01:28.494177 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.494159 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" Nov 03 20:01:28.494294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.494282 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" containerName="istio-proxy" Nov 03 20:01:28.494398 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.494387 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1e9cb4-5a73-43ce-bac0-a56b36bb3f8a" containerName="revoke-keys" Nov 03 20:01:28.515794 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.515762 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv"] Nov 03 20:01:28.515941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.515877 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.600848 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-envoy\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.600968 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-istio-token\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-istio-token\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.601029 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-certs\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.601060 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99c93810-f8d3-41c6-b220-74c97629b25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c93810-f8d3-41c6-b220-74c97629b25a-istio-podinfo\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.601381 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-data\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.601412 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-socket\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.601450 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c93810-f8d3-41c6-b220-74c97629b25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c93810-f8d3-41c6-b220-74c97629b25a-istiod-ca-cert\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.601480 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-kube-api-access-7gft7\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-kube-api-access-7gft7\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.603908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.601542 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-credential-socket\") pod \"99c93810-f8d3-41c6-b220-74c97629b25a\" (UID: \"99c93810-f8d3-41c6-b220-74c97629b25a\") " Nov 03 20:01:28.606057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.605763 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c93810-f8d3-41c6-b220-74c97629b25a-istiod-ca-cert" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:28.607351 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.607017 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-data" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:28.607351 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.607184 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-certs" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:28.607351 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.607330 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-socket" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:28.607784 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.607764 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-credential-socket" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:28.611998 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.611972 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2747a4-serving-cert\"" Nov 03 20:01:28.620030 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.619991 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99c93810-f8d3-41c6-b220-74c97629b25a-istio-podinfo" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:01:28.621382 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.621345 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-istio-token" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:28.623404 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.623376 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-envoy" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:28.633958 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.633926 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-kube-api-access-7gft7" pod "99c93810-f8d3-41c6-b220-74c97629b25a" (UID: "99c93810-f8d3-41c6-b220-74c97629b25a"). InnerVolumeSpecName "kube-api-access-7gft7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:28.704634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.704499 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.704800 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.704707 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61595c22-fea0-4971-82f6-47ca7990843e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.704878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.704859 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-metrics-certs\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.705124 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705103 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwdm\" (UniqueName: \"kubernetes.io/projected/61595c22-fea0-4971-82f6-47ca7990843e-kube-api-access-hjwdm\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.705413 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705397 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705471 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705419 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705471 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705434 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705471 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705447 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c93810-f8d3-41c6-b220-74c97629b25a-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705471 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705461 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705476 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705490 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c93810-f8d3-41c6-b220-74c97629b25a-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705503 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gft7\" (UniqueName: \"kubernetes.io/projected/99c93810-f8d3-41c6-b220-74c97629b25a-kube-api-access-7gft7\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.705714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.705517 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c93810-f8d3-41c6-b220-74c97629b25a-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:28.806647 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.806572 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.806647 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.806638 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61595c22-fea0-4971-82f6-47ca7990843e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.806881 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.806682 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-metrics-certs\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.806946 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.806906 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwdm\" (UniqueName: \"kubernetes.io/projected/61595c22-fea0-4971-82f6-47ca7990843e-kube-api-access-hjwdm\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.807497 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.807470 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61595c22-fea0-4971-82f6-47ca7990843e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.809263 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.809242 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-maas-api-tls\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.809263 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.809264 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-metrics-certs\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.814816 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.814797 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwdm\" (UniqueName: \"kubernetes.io/projected/61595c22-fea0-4971-82f6-47ca7990843e-kube-api-access-hjwdm\") pod \"maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:28.917606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:28.917564 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:29.117191 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.117147 4314 generic.go:356] "Generic (PLEG): container finished" podID="99c93810-f8d3-41c6-b220-74c97629b25a" containerID="4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953" exitCode=0 Nov 03 20:01:29.117365 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.117207 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" Nov 03 20:01:29.117439 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.117403 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" event={"ID":"99c93810-f8d3-41c6-b220-74c97629b25a","Type":"ContainerDied","Data":"4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953"} Nov 03 20:01:29.117509 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.117492 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv" event={"ID":"99c93810-f8d3-41c6-b220-74c97629b25a","Type":"ContainerDied","Data":"3c78a81572bc0ba4f0f982e32074cb37f79c8b2574ae07937337db03aded2c5d"} Nov 03 20:01:29.117547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.117533 4314 scope.go:122] "RemoveContainer" containerID="4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953" Nov 03 20:01:29.130488 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.130471 4314 scope.go:122] "RemoveContainer" containerID="9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040" Nov 03 20:01:29.152272 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.152249 4314 scope.go:122] "RemoveContainer" containerID="4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953" Nov 03 20:01:29.153446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.153387 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv"] Nov 03 20:01:29.156323 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:01:29.156262 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953\": container with ID starting with 4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953 not found: ID does not exist" containerID="4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953" Nov 03 20:01:29.156323 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.156302 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953"} err="failed to get container status \"4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953\": rpc error: code = NotFound desc = could not find container \"4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953\": container with ID starting with 4aa38746287dcf1c0f3a4c9ea8a27e5592391246a3ceb7ca0db3e43485913953 not found: ID does not exist" Nov 03 20:01:29.156502 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.156329 4314 scope.go:122] "RemoveContainer" containerID="9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040" Nov 03 20:01:29.156876 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:01:29.156848 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040\": container with ID starting with 9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040 not found: ID does not exist" containerID="9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040" Nov 03 20:01:29.156986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.156884 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040"} err="failed to get container status \"9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040\": rpc error: code = NotFound desc = could not find container \"9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040\": container with ID starting with 9e24521a435a17af370e03b3e784030a1f02af9f107d7ce52fe2691c60367040 not found: ID does not exist" Nov 03 20:01:29.159858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.159835 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bdb694-openshift-default-5698cb5d69-rfjmv"] Nov 03 20:01:29.208506 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.208475 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv"] Nov 03 20:01:29.208893 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:01:29.208870 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61595c22_fea0_4971_82f6_47ca7990843e.slice/crio-b4f0d3868f65660103603dc7836b5e69a357ed63650bf4ada6c64ea0c7dffd4c WatchSource:0}: Error finding container b4f0d3868f65660103603dc7836b5e69a357ed63650bf4ada6c64ea0c7dffd4c: Status 404 returned error can't find the container with id b4f0d3868f65660103603dc7836b5e69a357ed63650bf4ada6c64ea0c7dffd4c Nov 03 20:01:29.210366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:29.210347 4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:01:30.128656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:30.128619 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" event={"ID":"61595c22-fea0-4971-82f6-47ca7990843e","Type":"ContainerStarted","Data":"9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc"} Nov 03 20:01:30.129100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:30.128677 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" event={"ID":"61595c22-fea0-4971-82f6-47ca7990843e","Type":"ContainerStarted","Data":"b4f0d3868f65660103603dc7836b5e69a357ed63650bf4ada6c64ea0c7dffd4c"} Nov 03 20:01:30.129100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:30.128755 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:30.157126 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:30.157063 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" podStartSLOduration=212.728154309 podStartE2EDuration="3m33.157044994s" totalImagesPullingTime="428.890685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:30.150270616 +0000 UTC m=+1797.855109958" watchObservedRunningTime="2026-11-03 20:01:30.157044994 +0000 UTC m=+1797.861884335" Nov 03 20:01:30.431795 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:30.431720 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c93810-f8d3-41c6-b220-74c97629b25a" path="/var/lib/kubelet/pods/99c93810-f8d3-41c6-b220-74c97629b25a/volumes" Nov 03 20:01:33.115675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:33.115644 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:01:33.119156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:33.119128 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:01:36.179078 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:36.179048 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:01:42.449253 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:42.449214 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:01:49.301047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:49.301009 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth"] Nov 03 20:01:49.301527 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:49.301267 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" containerID="cri-o://7842ce2dcee25666ac21904b78eca3b756d16fcd29413757b415a7ad9f5d31ed" gracePeriod=30 Nov 03 20:01:49.746233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:49.746198 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d"] Nov 03 20:01:49.746710 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:49.746658 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" containerID="cri-o://489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a" gracePeriod=30 Nov 03 20:01:51.685337 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:51.685293 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th"] Nov 03 20:01:51.685887 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:51.685553 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" podUID="dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" containerName="maas-api" containerID="cri-o://775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb" gracePeriod=30 Nov 03 20:01:52.415882 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:01:52.412109 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:01:52.471634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.471577 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 20:01:52.528569 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.528531 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-kube-api-access-jq2cf\" (UniqueName: \"kubernetes.io/projected/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-kube-api-access-jq2cf\") pod \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " Nov 03 20:01:52.528775 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.528625 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-maas-api-tls\") pod \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " Nov 03 20:01:52.528775 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.528694 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-metrics-certs\") pod \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " Nov 03 20:01:52.529040 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.529018 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-openshift-service-ca\") pod \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\" (UID: \"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3\") " Nov 03 20:01:52.531874 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.531841 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-openshift-service-ca" pod "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" (UID: "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:52.532999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.532974 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-kube-api-access-jq2cf" pod "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" (UID: "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3"). InnerVolumeSpecName "kube-api-access-jq2cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:52.535028 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.534695 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-maas-api-tls" pod "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" (UID: "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:52.535028 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.534993 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-metrics-certs" pod "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" (UID: "dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:52.542616 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.542558 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq"] Nov 03 20:01:52.544046 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.544026 4314 cpu_manager.go:414] "containerMap: removing container" podUID="dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" containerName="maas-api" Nov 03 20:01:52.544104 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.544050 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" containerName="maas-api" Nov 03 20:01:52.544258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.544245 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" containerName="maas-api" Nov 03 20:01:52.549089 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.549070 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.550293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.550262 4314 generic.go:356] "Generic (PLEG): container finished" podID="dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" containerID="775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb" exitCode=0 Nov 03 20:01:52.550293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.550292 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" Nov 03 20:01:52.550476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.550359 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" event={"ID":"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3","Type":"ContainerDied","Data":"775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb"} Nov 03 20:01:52.550476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.550423 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th" event={"ID":"dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3","Type":"ContainerDied","Data":"cb7ea86f636b357859d72b01253adde652f566f1d8e1dbceabed94eb4673b1d0"} Nov 03 20:01:52.550703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.550480 4314 scope.go:122] "RemoveContainer" containerID="775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb" Nov 03 20:01:52.550703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.550660 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq"] Nov 03 20:01:52.594066 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.594038 4314 scope.go:122] "RemoveContainer" containerID="775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb" Nov 03 20:01:52.594702 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:01:52.594677 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb\": container with ID starting with 775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb not found: ID does not exist" containerID="775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb" Nov 03 20:01:52.594817 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.594711 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb"} err="failed to get container status \"775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb\": rpc error: code = NotFound desc = could not find container \"775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb\": container with ID starting with 775b02a420f559554f6e7c3301ff6a755ce12b3c763b44e94a0d95b64b08bbfb not found: ID does not exist" Nov 03 20:01:52.611749 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.611122 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th"] Nov 03 20:01:52.617155 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.617122 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-332694-d77564d9b-9n9th"] Nov 03 20:01:52.632720 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.632631 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4f7\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-kube-api-access-5p4f7\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.632720 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.632694 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba5076-0c89-43ed-a408-0005716f7a87-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.632937 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.632887 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.633163 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633141 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.633216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633200 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba5076-0c89-43ed-a408-0005716f7a87-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.633391 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633376 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.633451 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633436 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.633490 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633482 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.633524 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633514 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.635665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633728 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq2cf\" (UniqueName: \"kubernetes.io/projected/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-kube-api-access-jq2cf\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:52.635665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633754 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:52.635665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633777 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:52.635665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.633794 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:52.639094 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.639059 4314 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:01:52.639272 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.639128 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:01:52.735072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.734975 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735041 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba5076-0c89-43ed-a408-0005716f7a87-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735232 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735284 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735324 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735357 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735404 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4f7\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-kube-api-access-5p4f7\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735450 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba5076-0c89-43ed-a408-0005716f7a87-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735513 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735631 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735656 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.735979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.735849 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba5076-0c89-43ed-a408-0005716f7a87-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.736153 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.736000 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.736153 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.736088 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.737753 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.737730 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.738197 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.738177 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba5076-0c89-43ed-a408-0005716f7a87-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.743541 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.743485 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.743769 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.743721 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4f7\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-kube-api-access-5p4f7\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:52.903734 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:52.903690 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:53.211278 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:53.211247 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq"] Nov 03 20:01:53.211415 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:01:53.211244 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ba5076_0c89_43ed_a408_0005716f7a87.slice/crio-ebe9c30e23bbc38f416bb16515ce497cf6032ee28e40ebcf950cd2c2f470f38d WatchSource:0}: Error finding container ebe9c30e23bbc38f416bb16515ce497cf6032ee28e40ebcf950cd2c2f470f38d: Status 404 returned error can't find the container with id ebe9c30e23bbc38f416bb16515ce497cf6032ee28e40ebcf950cd2c2f470f38d Nov 03 20:01:53.214532 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:53.214493 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:01:53.214666 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:53.214565 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:01:53.214666 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:53.214651 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:01:53.562103 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:53.562062 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" event={"ID":"40ba5076-0c89-43ed-a408-0005716f7a87","Type":"ContainerStarted","Data":"e0871c6f718aa1ed48432c7d8442606c730187cd81269a9e217bf27d2d53f741"} Nov 03 20:01:53.562309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:53.562128 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" event={"ID":"40ba5076-0c89-43ed-a408-0005716f7a87","Type":"ContainerStarted","Data":"ebe9c30e23bbc38f416bb16515ce497cf6032ee28e40ebcf950cd2c2f470f38d"} Nov 03 20:01:53.904138 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:53.904058 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:01:54.516617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.504457 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3" path="/var/lib/kubelet/pods/dcbf8236-9f6e-4c50-a6ad-e99cf2bdb8b3/volumes" Nov 03 20:01:54.608653 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.605832 4314 generic.go:356] "Generic (PLEG): container finished" podID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerID="7842ce2dcee25666ac21904b78eca3b756d16fcd29413757b415a7ad9f5d31ed" exitCode=0 Nov 03 20:01:54.608653 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.605919 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" event={"ID":"bf55eb93-f7cf-45ce-aff9-b0af32b48eef","Type":"ContainerDied","Data":"7842ce2dcee25666ac21904b78eca3b756d16fcd29413757b415a7ad9f5d31ed"} Nov 03 20:01:54.819638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.819613 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 20:01:54.844905 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.844825 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podStartSLOduration=209.844708707 podStartE2EDuration="3m29.844708707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:53.589631793 +0000 UTC m=+1821.294471133" watchObservedRunningTime="2026-11-03 20:01:54.844708707 +0000 UTC m=+1822.549548047" Nov 03 20:01:54.916547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.914899 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp"] Nov 03 20:01:54.916547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.915886 4314 cpu_manager.go:414] "containerMap: removing container" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" Nov 03 20:01:54.916547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.915909 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" Nov 03 20:01:54.916547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.916135 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" containerName="istio-proxy" Nov 03 20:01:54.917067 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.916769 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:01:54.917067 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.916839 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 20:01:54.921612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.921554 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp"] Nov 03 20:01:54.922072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.922046 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:54.925373 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.925351 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-podinfo\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.925509 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.925495 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istiod-ca-cert\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.925661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.925645 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-token\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-token\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.925866 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.925851 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-data\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.926026 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.926013 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-envoy\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.926233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.926218 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-kube-api-access-rbm2s\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-kube-api-access-rbm2s\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.926461 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.926448 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-certs\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.926734 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.926719 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-socket\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.926843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.926830 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-credential-socket\") pod \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\" (UID: \"bf55eb93-f7cf-45ce-aff9-b0af32b48eef\") " Nov 03 20:01:54.928721 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.928694 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-credential-socket" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:54.928968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.928935 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-data" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:54.929510 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.929482 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-certs" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:54.929715 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.929692 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-socket" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:54.930166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.930140 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istiod-ca-cert" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:54.931145 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.931119 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-token" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:54.932396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.932368 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-envoy" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:54.937001 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.936959 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-kube-api-access-rbm2s" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "kube-api-access-rbm2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:54.939626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:54.938727 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-podinfo" pod "bf55eb93-f7cf-45ce-aff9-b0af32b48eef" (UID: "bf55eb93-f7cf-45ce-aff9-b0af32b48eef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:01:55.031365 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031315 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db981728-088c-4efd-a9b8-6cdee6b42866-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.031512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031430 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.031512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031468 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgw5z\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-kube-api-access-kgw5z\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.031512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031506 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.031752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031549 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.031811 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031769 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.031881 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031861 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.031992 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.031935 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.032151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032132 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db981728-088c-4efd-a9b8-6cdee6b42866-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.032303 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032285 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032310 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032327 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032341 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032358 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032373 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbm2s\" (UniqueName: \"kubernetes.io/projected/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-kube-api-access-rbm2s\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032386 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032400 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.032570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.032418 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf55eb93-f7cf-45ce-aff9-b0af32b48eef-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135097 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135189 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135250 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135351 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db981728-088c-4efd-a9b8-6cdee6b42866-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135438 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db981728-088c-4efd-a9b8-6cdee6b42866-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135517 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135548 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgw5z\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-kube-api-access-kgw5z\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135594 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.136123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.135633 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.140941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.137195 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.140941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.137466 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.140941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.138126 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db981728-088c-4efd-a9b8-6cdee6b42866-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.140941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.138655 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.140941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.139019 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.140941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.139163 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.146870 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.143122 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db981728-088c-4efd-a9b8-6cdee6b42866-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.151456 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.151435 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 20:01:55.154291 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.154257 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.173013 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.167298 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgw5z\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-kube-api-access-kgw5z\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.237442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237246 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-credential-socket\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.237442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237420 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-certs\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.237717 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237498 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-data\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.237717 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237538 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-credential-socket" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:55.237717 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237711 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-envoy\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.237913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237753 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5607bf06-0901-4fcc-8914-5cd2e90212d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5607bf06-0901-4fcc-8914-5cd2e90212d8-istiod-ca-cert\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.237913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237836 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-socket\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.237913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237860 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-kube-api-access-879tb\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-kube-api-access-879tb\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.237913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237883 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-certs" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:55.237913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237907 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-podinfo\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.238213 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.237962 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-token\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-token\") pod \"5607bf06-0901-4fcc-8914-5cd2e90212d8\" (UID: \"5607bf06-0901-4fcc-8914-5cd2e90212d8\") " Nov 03 20:01:55.238213 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.238189 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5607bf06-0901-4fcc-8914-5cd2e90212d8-istiod-ca-cert" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:55.238671 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.238635 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-socket" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:55.239078 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.239054 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.239202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.239080 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.239202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.239097 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.239202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.239113 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5607bf06-0901-4fcc-8914-5cd2e90212d8-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.239202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.239081 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-data" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:55.240682 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.240657 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-kube-api-access-879tb" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "kube-api-access-879tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:55.240876 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.240855 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-podinfo" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:01:55.240933 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.240888 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-token" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:55.241002 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.240978 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-envoy" pod "5607bf06-0901-4fcc-8914-5cd2e90212d8" (UID: "5607bf06-0901-4fcc-8914-5cd2e90212d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:55.340663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.340615 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.340663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.340645 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-879tb\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-kube-api-access-879tb\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.340663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.340655 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.340663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.340665 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.341015 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.340675 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5607bf06-0901-4fcc-8914-5cd2e90212d8-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:55.343839 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.343810 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:55.620112 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.620082 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" Nov 03 20:01:55.622808 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.620851 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth" event={"ID":"bf55eb93-f7cf-45ce-aff9-b0af32b48eef","Type":"ContainerDied","Data":"42d3a82ccf8f66ce6c93214d6782715dcfedcbe6c7eb06d03ffbc08c6b3a06ee"} Nov 03 20:01:55.622808 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.620919 4314 scope.go:122] "RemoveContainer" containerID="7842ce2dcee25666ac21904b78eca3b756d16fcd29413757b415a7ad9f5d31ed" Nov 03 20:01:55.623312 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.623285 4314 generic.go:356] "Generic (PLEG): container finished" podID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerID="489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a" exitCode=0 Nov 03 20:01:55.623767 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.623749 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" Nov 03 20:01:55.624025 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.624005 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" event={"ID":"5607bf06-0901-4fcc-8914-5cd2e90212d8","Type":"ContainerDied","Data":"489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a"} Nov 03 20:01:55.624161 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.624147 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d" event={"ID":"5607bf06-0901-4fcc-8914-5cd2e90212d8","Type":"ContainerDied","Data":"dfacbf6782bd71e6bb0e091919e8cf028584f7e568c48b5770a4ca8074a2bf9d"} Nov 03 20:01:55.635825 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.635805 4314 scope.go:122] "RemoveContainer" containerID="489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a" Nov 03 20:01:55.660170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.659781 4314 scope.go:122] "RemoveContainer" containerID="489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a" Nov 03 20:01:55.660667 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:01:55.660472 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a\": container with ID starting with 489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a not found: ID does not exist" containerID="489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a" Nov 03 20:01:55.660667 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.660516 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a"} err="failed to get container status \"489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a\": rpc error: code = NotFound desc = could not find container \"489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a\": container with ID starting with 489c4403a21076141b8fc4eb41261b2bfde780485a855b8fb7625ccd8a3d842a not found: ID does not exist" Nov 03 20:01:55.712078 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.705660 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth"] Nov 03 20:01:55.726804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.725515 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-c4rth"] Nov 03 20:01:55.736946 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.736921 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d"] Nov 03 20:01:55.745727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.745695 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-h4b5d"] Nov 03 20:01:55.897368 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:01:55.897319 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb981728_088c_4efd_a9b8_6cdee6b42866.slice/crio-528fffe561834249b4bcca32d31063bb3e7f6b75551f8468818789b84ebaf201 WatchSource:0}: Error finding container 528fffe561834249b4bcca32d31063bb3e7f6b75551f8468818789b84ebaf201: Status 404 returned error can't find the container with id 528fffe561834249b4bcca32d31063bb3e7f6b75551f8468818789b84ebaf201 Nov 03 20:01:55.901421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.901382 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:01:55.901534 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.901478 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:01:55.901534 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.901527 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:01:55.907517 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.907488 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp"] Nov 03 20:01:55.910735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.910708 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:01:55.913963 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:55.912675 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 20:01:56.431561 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:56.431529 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" path="/var/lib/kubelet/pods/5607bf06-0901-4fcc-8914-5cd2e90212d8/volumes" Nov 03 20:01:56.432196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:56.432178 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf55eb93-f7cf-45ce-aff9-b0af32b48eef" path="/var/lib/kubelet/pods/bf55eb93-f7cf-45ce-aff9-b0af32b48eef/volumes" Nov 03 20:01:56.637857 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:56.637824 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" event={"ID":"db981728-088c-4efd-a9b8-6cdee6b42866","Type":"ContainerStarted","Data":"9f3633a327e4aadacde7aea34daa2cb7c66d41a7006445e9397d5720e9b25513"} Nov 03 20:01:56.637999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:56.637875 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" event={"ID":"db981728-088c-4efd-a9b8-6cdee6b42866","Type":"ContainerStarted","Data":"528fffe561834249b4bcca32d31063bb3e7f6b75551f8468818789b84ebaf201"} Nov 03 20:01:56.710298 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:56.706770 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podStartSLOduration=211.706750051 podStartE2EDuration="3m31.706750051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:56.67829582 +0000 UTC m=+1824.383135172" watchObservedRunningTime="2026-11-03 20:01:56.706750051 +0000 UTC m=+1824.411589394" Nov 03 20:01:56.904954 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:56.904897 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:01:56.905159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:56.904984 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 20:01:57.344417 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:57.344380 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:01:57.345683 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:57.345654 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:01:57.345820 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:57.345705 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:01:57.905870 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:57.905236 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:01:57.905870 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:57.905301 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 20:01:58.344987 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:58.344957 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:01:58.345213 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:58.345191 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:01:58.910735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:58.910001 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:01:58.910735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:58.910066 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 20:01:59.570605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:59.520715 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:01:59.570605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:59.520792 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:01:59.589224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:59.571948 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz"] Nov 03 20:01:59.589224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:59.572354 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" containerID="cri-o://6f4d0dbe69a2d97b10958bcce8dd8ee4fa2a8ad8654cba4b5277030ec2e28583" gracePeriod=30 Nov 03 20:01:59.972672 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:59.970751 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr"] Nov 03 20:01:59.972672 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:01:59.971005 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" containerID="cri-o://50f8b5c2ad68c01035dabea00a0d84f0ec2201b677a2eee0a26e437a51f81b51" gracePeriod=30 Nov 03 20:02:00.028620 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.020600 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:02:00.028620 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.020666 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 20:02:00.055640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.055603 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g"] Nov 03 20:02:00.055893 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.055867 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" containerID="cri-o://91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862" gracePeriod=30 Nov 03 20:02:00.137493 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.135697 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:02:00.345623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.345511 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:02:00.345850 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.345631 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:02:00.741526 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.741500 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp"] Nov 03 20:02:00.742888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.742855 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" Nov 03 20:02:00.743034 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.743020 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" Nov 03 20:02:00.743390 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.743378 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5607bf06-0901-4fcc-8914-5cd2e90212d8" containerName="istio-proxy" Nov 03 20:02:00.757945 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.757916 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.783829 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.754851 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp"] Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.825943 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-3ff278-openshift-default-dockercfg-c45jl\"" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827220 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-credential-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827279 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istiod-ca-cert\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827333 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-envoy\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827599 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-data\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827644 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827685 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-token\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827768 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-certs\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827856 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7vc\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-kube-api-access-vd7vc\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.839621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.827942 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-podinfo\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.954799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.954761 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-data\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.996997 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.995259 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.996997 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.995318 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-token\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.996997 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.995411 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-certs\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.996997 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.995501 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7vc\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-kube-api-access-vd7vc\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:00.998066 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.970889 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:02:00.998287 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.998246 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 20:02:00.998428 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:00.955948 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-data\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.014703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.010811 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-podinfo\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.014703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.011015 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-credential-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.014703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.011078 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istiod-ca-cert\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.014703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.011138 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-envoy\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.015838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.015405 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-credential-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.027140 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.017651 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-socket\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.027140 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.017918 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-certs\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.027681 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.027571 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istiod-ca-cert\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.105729 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.096488 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-podinfo\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.105729 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.096893 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-envoy\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.122606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.119367 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-token\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.151139 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.127442 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7vc\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-kube-api-access-vd7vc\") pod \"e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:01.344679 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.344560 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:02:01.344843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.344672 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:02:01.411605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:01.407109 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:02.048990 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.040963 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:02.048990 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.041060 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:02.048990 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.041109 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:02.222377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.222334 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp"] Nov 03 20:02:02.241931 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.241899 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" event={"ID":"1fedb898-5b21-48a4-bf0d-3f603e9a717b","Type":"ContainerStarted","Data":"dc1bca63b84ddc68fd27ef273daef34e52359394dc08778ee163bb5bc22ebb3e"} Nov 03 20:02:02.323620 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.320689 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:02:02.323620 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.321460 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:02:02.323620 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.322969 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:02:02.346635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.346530 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:02:02.346635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:02.346608 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:02:03.254007 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:03.253971 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" event={"ID":"1fedb898-5b21-48a4-bf0d-3f603e9a717b","Type":"ContainerStarted","Data":"7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a"} Nov 03 20:02:03.292750 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:03.292664 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podStartSLOduration=3.292643308 podStartE2EDuration="3.292643308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:03.27994506 +0000 UTC m=+1830.984784403" watchObservedRunningTime="2026-11-03 20:02:03.292643308 +0000 UTC m=+1830.997482649" Nov 03 20:02:03.344754 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:03.344723 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:02:03.344916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:03.344780 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:02:03.421607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:03.412708 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:03.497607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:03.495775 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:04.292606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:04.292114 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:04.294655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:04.294145 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:02:04.344806 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:04.344771 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:02:04.344965 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:04.344828 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 20:02:05.325065 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.324647 4314 generic.go:356] "Generic (PLEG): container finished" podID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerID="6f4d0dbe69a2d97b10958bcce8dd8ee4fa2a8ad8654cba4b5277030ec2e28583" exitCode=0 Nov 03 20:02:05.325065 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.324905 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" event={"ID":"98e1c8f5-9908-4067-818e-98a7a90eebbf","Type":"ContainerDied","Data":"6f4d0dbe69a2d97b10958bcce8dd8ee4fa2a8ad8654cba4b5277030ec2e28583"} Nov 03 20:02:05.329291 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.327037 4314 generic.go:356] "Generic (PLEG): container finished" podID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerID="50f8b5c2ad68c01035dabea00a0d84f0ec2201b677a2eee0a26e437a51f81b51" exitCode=0 Nov 03 20:02:05.329291 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.327197 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" event={"ID":"e7bdfc8b-2b4e-4646-8381-7166c60090b9","Type":"ContainerDied","Data":"50f8b5c2ad68c01035dabea00a0d84f0ec2201b677a2eee0a26e437a51f81b51"} Nov 03 20:02:05.359170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.353026 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:02:05.359170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.355003 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:02:05.359170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.355873 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:02:05.443116 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.442776 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 20:02:05.451622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.450606 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524181 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-socket\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524225 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-certs\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524270 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-envoy\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524315 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-token\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-token\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524358 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-credential-socket\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524390 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istiod-ca-cert\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524427 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-podinfo\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524475 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-data\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.524563 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-data\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.525429 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istiod-ca-cert" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.525980 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-credential-socket" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.526802 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-certs" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.527093 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-socket" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.527295 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-data" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.527485 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-data" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.527858 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-credential-socket\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.527987 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e1c8f5-9908-4067-818e-98a7a90eebbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e1c8f5-9908-4067-818e-98a7a90eebbf-istiod-ca-cert\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.528028 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-envoy\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.528139 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-kube-api-access-t429v\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-kube-api-access-t429v\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.528186 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-certs\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.528260 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-podinfo\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.528343 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-socket\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.528476 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-kube-api-access-5cw9m\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-kube-api-access-5cw9m\") pod \"98e1c8f5-9908-4067-818e-98a7a90eebbf\" (UID: \"98e1c8f5-9908-4067-818e-98a7a90eebbf\") " Nov 03 20:02:05.528668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.528548 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-token\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-token\") pod \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\" (UID: \"e7bdfc8b-2b4e-4646-8381-7166c60090b9\") " Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.532784 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.533352 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.533373 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.533389 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.534345 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.534369 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.540080 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e1c8f5-9908-4067-818e-98a7a90eebbf-istiod-ca-cert" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.540375 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-credential-socket" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.547397 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-certs" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.560350 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-podinfo" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.562053 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-envoy" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.564442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.562773 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-socket" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.581837 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.581786 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-token" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:05.582247 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.582208 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-kube-api-access-5cw9m" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "kube-api-access-5cw9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:05.582545 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.582505 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-token" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:05.582545 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.582511 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-envoy" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.585423 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.585015 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-podinfo" pod "98e1c8f5-9908-4067-818e-98a7a90eebbf" (UID: "98e1c8f5-9908-4067-818e-98a7a90eebbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:05.586028 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.585997 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-kube-api-access-t429v" pod "e7bdfc8b-2b4e-4646-8381-7166c60090b9" (UID: "e7bdfc8b-2b4e-4646-8381-7166c60090b9"). InnerVolumeSpecName "kube-api-access-t429v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:05.637236 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637193 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637236 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637236 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cw9m\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-kube-api-access-5cw9m\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637253 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637266 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637280 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637292 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98e1c8f5-9908-4067-818e-98a7a90eebbf-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637306 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637320 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98e1c8f5-9908-4067-818e-98a7a90eebbf-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637335 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637348 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t429v\" (UniqueName: \"kubernetes.io/projected/e7bdfc8b-2b4e-4646-8381-7166c60090b9-kube-api-access-t429v\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637361 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98e1c8f5-9908-4067-818e-98a7a90eebbf-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.637482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.637375 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7bdfc8b-2b4e-4646-8381-7166c60090b9-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.684602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.682703 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 20:02:05.738719 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.738672 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-istio-token\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-istio-token\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.738884 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.738807 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-envoy\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.738969 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.738944 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-certs\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.739120 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.739097 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-socket\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.739195 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.739173 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021031e4-06e6-4109-bea5-44b520187c94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021031e4-06e6-4109-bea5-44b520187c94-istiod-ca-cert\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.739343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.739318 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-credential-socket\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.739626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.739602 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/021031e4-06e6-4109-bea5-44b520187c94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021031e4-06e6-4109-bea5-44b520187c94-istio-podinfo\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.739727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.739683 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-data\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.739789 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.739737 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-kube-api-access-sgbf9\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-kube-api-access-sgbf9\") pod \"021031e4-06e6-4109-bea5-44b520187c94\" (UID: \"021031e4-06e6-4109-bea5-44b520187c94\") " Nov 03 20:02:05.739879 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.739841 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-certs" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.741181 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.741153 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.743059 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.743030 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-istio-token" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:05.744967 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.744938 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-envoy" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.745074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.745040 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-data" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.745135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.745050 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-socket" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.745286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.745266 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021031e4-06e6-4109-bea5-44b520187c94-istiod-ca-cert" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:05.746806 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.746779 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-credential-socket" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:05.747520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.747499 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-kube-api-access-sgbf9" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "kube-api-access-sgbf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:05.748632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.748603 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/021031e4-06e6-4109-bea5-44b520187c94-istio-podinfo" pod "021031e4-06e6-4109-bea5-44b520187c94" (UID: "021031e4-06e6-4109-bea5-44b520187c94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:05.842449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842374 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021031e4-06e6-4109-bea5-44b520187c94-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.842449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842402 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.842449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842411 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbf9\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-kube-api-access-sgbf9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.842449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842421 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021031e4-06e6-4109-bea5-44b520187c94-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.842449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842429 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.842449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842438 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.842449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842446 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021031e4-06e6-4109-bea5-44b520187c94-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:05.842822 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:05.842455 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021031e4-06e6-4109-bea5-44b520187c94-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:06.336868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.336834 4314 generic.go:356] "Generic (PLEG): container finished" podID="021031e4-06e6-4109-bea5-44b520187c94" containerID="91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862" exitCode=0 Nov 03 20:02:06.337315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.336895 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" Nov 03 20:02:06.337315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.336926 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" event={"ID":"021031e4-06e6-4109-bea5-44b520187c94","Type":"ContainerDied","Data":"91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862"} Nov 03 20:02:06.337315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.337012 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g" event={"ID":"021031e4-06e6-4109-bea5-44b520187c94","Type":"ContainerDied","Data":"9bdb86c1ee96469406ba8484f8f57058c924a9bd39e71b0ca722a47eaca4b737"} Nov 03 20:02:06.337315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.337043 4314 scope.go:122] "RemoveContainer" containerID="91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862" Nov 03 20:02:06.338462 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.338444 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" Nov 03 20:02:06.338462 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.338451 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr" event={"ID":"e7bdfc8b-2b4e-4646-8381-7166c60090b9","Type":"ContainerDied","Data":"f0f1b65e21c35b243865d8210f13e05823bbc99e6e6a67d880b210568c8d7eaf"} Nov 03 20:02:06.340719 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.340683 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" event={"ID":"98e1c8f5-9908-4067-818e-98a7a90eebbf","Type":"ContainerDied","Data":"83c408f43e6f78c6df728068643cbedd3671f738885beb3872dbcb3723859b93"} Nov 03 20:02:06.340821 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.340743 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz" Nov 03 20:02:06.351375 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.351353 4314 scope.go:122] "RemoveContainer" containerID="5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8" Nov 03 20:02:06.364330 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.364305 4314 scope.go:122] "RemoveContainer" containerID="91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862" Nov 03 20:02:06.364637 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:06.364607 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862\": container with ID starting with 91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862 not found: ID does not exist" containerID="91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862" Nov 03 20:02:06.364732 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.364646 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862"} err="failed to get container status \"91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862\": rpc error: code = NotFound desc = could not find container \"91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862\": container with ID starting with 91a573e5e538fc9b2f2f919a3dbbce6b36cff822a55ef5570a74f5fe2f2ea862 not found: ID does not exist" Nov 03 20:02:06.364732 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.364667 4314 scope.go:122] "RemoveContainer" containerID="5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8" Nov 03 20:02:06.367018 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:06.366996 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8\": container with ID starting with 5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8 not found: ID does not exist" containerID="5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8" Nov 03 20:02:06.367135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.367023 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8"} err="failed to get container status \"5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8\": rpc error: code = NotFound desc = could not find container \"5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8\": container with ID starting with 5c79b6114e5cf35aa1738ae3e81a89de5e1292d99cf39bdc45fa0bc306e479e8 not found: ID does not exist" Nov 03 20:02:06.367135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.367044 4314 scope.go:122] "RemoveContainer" containerID="50f8b5c2ad68c01035dabea00a0d84f0ec2201b677a2eee0a26e437a51f81b51" Nov 03 20:02:06.374624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.374599 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g"] Nov 03 20:02:06.379527 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.379507 4314 scope.go:122] "RemoveContainer" containerID="6f4d0dbe69a2d97b10958bcce8dd8ee4fa2a8ad8654cba4b5277030ec2e28583" Nov 03 20:02:06.382335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.382308 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wsl7g"] Nov 03 20:02:06.389453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.389429 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr"] Nov 03 20:02:06.394141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.394121 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-wztdr"] Nov 03 20:02:06.436775 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.436741 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021031e4-06e6-4109-bea5-44b520187c94" path="/var/lib/kubelet/pods/021031e4-06e6-4109-bea5-44b520187c94/volumes" Nov 03 20:02:06.437349 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.437333 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" path="/var/lib/kubelet/pods/e7bdfc8b-2b4e-4646-8381-7166c60090b9/volumes" Nov 03 20:02:06.437691 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.437677 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz"] Nov 03 20:02:06.437738 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.437707 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-8qrgz"] Nov 03 20:02:06.625821 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.624902 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv"] Nov 03 20:02:06.632704 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:06.632669 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" podUID="61595c22-fea0-4971-82f6-47ca7990843e" containerName="maas-api" containerID="cri-o://9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc" gracePeriod=30 Nov 03 20:02:07.857712 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.857679 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:02:07.884417 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.884372 4314 generic.go:356] "Generic (PLEG): container finished" podID="61595c22-fea0-4971-82f6-47ca7990843e" containerID="9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc" exitCode=0 Nov 03 20:02:07.887364 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.887329 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" event={"ID":"61595c22-fea0-4971-82f6-47ca7990843e","Type":"ContainerDied","Data":"9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc"} Nov 03 20:02:07.887658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.887635 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" event={"ID":"61595c22-fea0-4971-82f6-47ca7990843e","Type":"ContainerDied","Data":"b4f0d3868f65660103603dc7836b5e69a357ed63650bf4ada6c64ea0c7dffd4c"} Nov 03 20:02:07.887830 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.887815 4314 scope.go:122] "RemoveContainer" containerID="9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc" Nov 03 20:02:07.916773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.916232 4314 scope.go:122] "RemoveContainer" containerID="9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc" Nov 03 20:02:07.917212 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:07.917181 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc\": container with ID starting with 9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc not found: ID does not exist" containerID="9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc" Nov 03 20:02:07.917365 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.917340 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc"} err="failed to get container status \"9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc\": rpc error: code = NotFound desc = could not find container \"9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc\": container with ID starting with 9c43c9d7418634a4800e129bf9b71e0558c64e4d12255bc15673ec291a69fbbc not found: ID does not exist" Nov 03 20:02:07.944863 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.929327 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-maas-api-tls\") pod \"61595c22-fea0-4971-82f6-47ca7990843e\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " Nov 03 20:02:07.949572 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.949533 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61595c22-fea0-4971-82f6-47ca7990843e-kube-api-access-hjwdm\" (UniqueName: \"kubernetes.io/projected/61595c22-fea0-4971-82f6-47ca7990843e-kube-api-access-hjwdm\") pod \"61595c22-fea0-4971-82f6-47ca7990843e\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " Nov 03 20:02:07.950009 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.949984 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-metrics-certs\") pod \"61595c22-fea0-4971-82f6-47ca7990843e\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " Nov 03 20:02:07.950305 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.950286 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61595c22-fea0-4971-82f6-47ca7990843e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61595c22-fea0-4971-82f6-47ca7990843e-openshift-service-ca\") pod \"61595c22-fea0-4971-82f6-47ca7990843e\" (UID: \"61595c22-fea0-4971-82f6-47ca7990843e\") " Nov 03 20:02:07.950687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.950659 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61595c22-fea0-4971-82f6-47ca7990843e-openshift-service-ca" pod "61595c22-fea0-4971-82f6-47ca7990843e" (UID: "61595c22-fea0-4971-82f6-47ca7990843e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:07.952254 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.952231 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61595c22-fea0-4971-82f6-47ca7990843e-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:07.952362 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.952331 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61595c22-fea0-4971-82f6-47ca7990843e-kube-api-access-hjwdm" pod "61595c22-fea0-4971-82f6-47ca7990843e" (UID: "61595c22-fea0-4971-82f6-47ca7990843e"). InnerVolumeSpecName "kube-api-access-hjwdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:07.953021 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.952991 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-maas-api-tls" pod "61595c22-fea0-4971-82f6-47ca7990843e" (UID: "61595c22-fea0-4971-82f6-47ca7990843e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:07.954417 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:07.954392 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-metrics-certs" pod "61595c22-fea0-4971-82f6-47ca7990843e" (UID: "61595c22-fea0-4971-82f6-47ca7990843e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:08.054499 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:08.054459 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.054801 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:08.054505 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjwdm\" (UniqueName: \"kubernetes.io/projected/61595c22-fea0-4971-82f6-47ca7990843e-kube-api-access-hjwdm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.054801 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:08.054525 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61595c22-fea0-4971-82f6-47ca7990843e-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.432522 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:08.432487 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" path="/var/lib/kubelet/pods/98e1c8f5-9908-4067-818e-98a7a90eebbf/volumes" Nov 03 20:02:08.896777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:08.896745 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv" Nov 03 20:02:08.917938 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:08.917909 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv"] Nov 03 20:02:08.922264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:08.922239 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2747a4-5c5c8df5f-vmgjv"] Nov 03 20:02:10.003447 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.002878 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f"] Nov 03 20:02:10.004175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004153 4314 cpu_manager.go:414] "containerMap: removing container" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" Nov 03 20:02:10.004264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004178 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" Nov 03 20:02:10.004264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004197 4314 cpu_manager.go:414] "containerMap: removing container" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" Nov 03 20:02:10.004264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004204 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" Nov 03 20:02:10.004264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004231 4314 cpu_manager.go:414] "containerMap: removing container" podUID="61595c22-fea0-4971-82f6-47ca7990843e" containerName="maas-api" Nov 03 20:02:10.004264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004238 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61595c22-fea0-4971-82f6-47ca7990843e" containerName="maas-api" Nov 03 20:02:10.004523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004296 4314 cpu_manager.go:414] "containerMap: removing container" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" Nov 03 20:02:10.004523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004304 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" Nov 03 20:02:10.004523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004516 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" Nov 03 20:02:10.004752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004534 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e1c8f5-9908-4067-818e-98a7a90eebbf" containerName="istio-proxy" Nov 03 20:02:10.004752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004555 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" Nov 03 20:02:10.004752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004566 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bdfc8b-2b4e-4646-8381-7166c60090b9" containerName="istio-proxy" Nov 03 20:02:10.004752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.004576 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="61595c22-fea0-4971-82f6-47ca7990843e" containerName="maas-api" Nov 03 20:02:10.010790 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.010763 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.015039 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.014995 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:10.016403 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.016380 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:02:10.018977 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.018933 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f"] Nov 03 20:02:10.111928 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.111885 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.112090 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.112034 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxwb\" (UniqueName: \"kubernetes.io/projected/85b8dc8c-de04-48d7-bfc2-87917a582601-kube-api-access-6vxwb\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.112268 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.112251 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.214194 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.214073 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.214406 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.214384 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.214476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.214458 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxwb\" (UniqueName: \"kubernetes.io/projected/85b8dc8c-de04-48d7-bfc2-87917a582601-kube-api-access-6vxwb\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.215231 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.215195 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.220971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.220946 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.222626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.222573 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxwb\" (UniqueName: \"kubernetes.io/projected/85b8dc8c-de04-48d7-bfc2-87917a582601-kube-api-access-6vxwb\") pod \"maas-api-revoke-sub-ec4d538f-qvq8f\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.328765 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.328685 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:10.467642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.462389 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61595c22-fea0-4971-82f6-47ca7990843e" path="/var/lib/kubelet/pods/61595c22-fea0-4971-82f6-47ca7990843e/volumes" Nov 03 20:02:10.637843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.637811 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f"] Nov 03 20:02:10.637961 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:02:10.637866 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b8dc8c_de04_48d7_bfc2_87917a582601.slice/crio-cbdfa3707e41f365075d8adbf61d6dad36b57566bc0defb30efb610203049eb9 WatchSource:0}: Error finding container cbdfa3707e41f365075d8adbf61d6dad36b57566bc0defb30efb610203049eb9: Status 404 returned error can't find the container with id cbdfa3707e41f365075d8adbf61d6dad36b57566bc0defb30efb610203049eb9 Nov 03 20:02:10.941689 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:10.941571 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" event={"ID":"85b8dc8c-de04-48d7-bfc2-87917a582601","Type":"ContainerStarted","Data":"cbdfa3707e41f365075d8adbf61d6dad36b57566bc0defb30efb610203049eb9"} Nov 03 20:02:11.951168 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:11.951132 4314 generic.go:356] "Generic (PLEG): container finished" podID="85b8dc8c-de04-48d7-bfc2-87917a582601" containerID="18742f29fe7d976a68ae0eee237ab337c3c2fe72724572a22998341563e4884f" exitCode=0 Nov 03 20:02:11.951551 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:11.951215 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" event={"ID":"85b8dc8c-de04-48d7-bfc2-87917a582601","Type":"ContainerDied","Data":"18742f29fe7d976a68ae0eee237ab337c3c2fe72724572a22998341563e4884f"} Nov 03 20:02:13.175803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.175778 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:13.177949 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.177922 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b8dc8c-de04-48d7-bfc2-87917a582601-kube-api-access-6vxwb\" (UniqueName: \"kubernetes.io/projected/85b8dc8c-de04-48d7-bfc2-87917a582601-kube-api-access-6vxwb\") pod \"85b8dc8c-de04-48d7-bfc2-87917a582601\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " Nov 03 20:02:13.178180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.178162 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-xks-ca\") pod \"85b8dc8c-de04-48d7-bfc2-87917a582601\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " Nov 03 20:02:13.178329 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.178311 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-openshift-ca\") pod \"85b8dc8c-de04-48d7-bfc2-87917a582601\" (UID: \"85b8dc8c-de04-48d7-bfc2-87917a582601\") " Nov 03 20:02:13.178785 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.178721 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-openshift-ca" pod "85b8dc8c-de04-48d7-bfc2-87917a582601" (UID: "85b8dc8c-de04-48d7-bfc2-87917a582601"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:13.179419 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.179394 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:13.180632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.180595 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-xks-ca" pod "85b8dc8c-de04-48d7-bfc2-87917a582601" (UID: "85b8dc8c-de04-48d7-bfc2-87917a582601"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:13.180764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.180665 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b8dc8c-de04-48d7-bfc2-87917a582601-kube-api-access-6vxwb" pod "85b8dc8c-de04-48d7-bfc2-87917a582601" (UID: "85b8dc8c-de04-48d7-bfc2-87917a582601"). InnerVolumeSpecName "kube-api-access-6vxwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:13.281820 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.281782 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85b8dc8c-de04-48d7-bfc2-87917a582601-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:13.281820 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.281819 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vxwb\" (UniqueName: \"kubernetes.io/projected/85b8dc8c-de04-48d7-bfc2-87917a582601-kube-api-access-6vxwb\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:13.972422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.972391 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" Nov 03 20:02:13.972609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.972391 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f" event={"ID":"85b8dc8c-de04-48d7-bfc2-87917a582601","Type":"ContainerDied","Data":"cbdfa3707e41f365075d8adbf61d6dad36b57566bc0defb30efb610203049eb9"} Nov 03 20:02:13.972609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:13.972549 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbdfa3707e41f365075d8adbf61d6dad36b57566bc0defb30efb610203049eb9" Nov 03 20:02:14.522670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:14.519649 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x"] Nov 03 20:02:14.522670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:14.519921 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" containerID="cri-o://1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547" gracePeriod=30 Nov 03 20:02:16.475377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.475337 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg"] Nov 03 20:02:16.482133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.481969 4314 cpu_manager.go:414] "containerMap: removing container" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" Nov 03 20:02:16.482133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.482000 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021031e4-06e6-4109-bea5-44b520187c94" containerName="istio-proxy" Nov 03 20:02:16.482133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.482017 4314 cpu_manager.go:414] "containerMap: removing container" podUID="85b8dc8c-de04-48d7-bfc2-87917a582601" containerName="revoke-keys" Nov 03 20:02:16.482133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.482024 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b8dc8c-de04-48d7-bfc2-87917a582601" containerName="revoke-keys" Nov 03 20:02:16.482478 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.482253 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b8dc8c-de04-48d7-bfc2-87917a582601" containerName="revoke-keys" Nov 03 20:02:16.487531 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.487490 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg"] Nov 03 20:02:16.487681 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.487620 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.492713 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.492692 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:02:16.492836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.492734 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:16.528692 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.528543 4314 patch_prober.go:29] interesting pod/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:02:16.528896 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.528866 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:02:16.565626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.564178 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.565880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.565854 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.574601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.565991 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfvpg\" (UniqueName: \"kubernetes.io/projected/d5e67e41-fb69-412b-aa31-03c50976ef72-kube-api-access-cfvpg\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.676657 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.676608 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.676838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.676680 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.676838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.676712 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfvpg\" (UniqueName: \"kubernetes.io/projected/d5e67e41-fb69-412b-aa31-03c50976ef72-kube-api-access-cfvpg\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.677829 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.677800 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.687041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.687002 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.690234 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.690207 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfvpg\" (UniqueName: \"kubernetes.io/projected/d5e67e41-fb69-412b-aa31-03c50976ef72-kube-api-access-cfvpg\") pod \"maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:16.827229 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:16.827198 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:17.124693 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:02:17.124661 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e67e41_fb69_412b_aa31_03c50976ef72.slice/crio-8b32f7ace3417f6710f1d21f8405c9047201c9b8ea21f19f84cfab0625abd3d8 WatchSource:0}: Error finding container 8b32f7ace3417f6710f1d21f8405c9047201c9b8ea21f19f84cfab0625abd3d8: Status 404 returned error can't find the container with id 8b32f7ace3417f6710f1d21f8405c9047201c9b8ea21f19f84cfab0625abd3d8 Nov 03 20:02:17.126406 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:17.126379 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg"] Nov 03 20:02:18.029891 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:18.029847 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" event={"ID":"d5e67e41-fb69-412b-aa31-03c50976ef72","Type":"ContainerStarted","Data":"101e06242ccc75924374ed00749d7cf16498415f960a88dfac51312c01437770"} Nov 03 20:02:18.030274 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:18.029919 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" event={"ID":"d5e67e41-fb69-412b-aa31-03c50976ef72","Type":"ContainerStarted","Data":"8b32f7ace3417f6710f1d21f8405c9047201c9b8ea21f19f84cfab0625abd3d8"} Nov 03 20:02:18.046909 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:18.046858 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" podStartSLOduration=2.046845986 podStartE2EDuration="2.046845986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:18.045431005 +0000 UTC m=+1845.750270347" watchObservedRunningTime="2026-11-03 20:02:18.046845986 +0000 UTC m=+1845.751685326" Nov 03 20:02:19.039419 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.039384 4314 generic.go:356] "Generic (PLEG): container finished" podID="d5e67e41-fb69-412b-aa31-03c50976ef72" containerID="101e06242ccc75924374ed00749d7cf16498415f960a88dfac51312c01437770" exitCode=0 Nov 03 20:02:19.039860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.039472 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" event={"ID":"d5e67e41-fb69-412b-aa31-03c50976ef72","Type":"ContainerDied","Data":"101e06242ccc75924374ed00749d7cf16498415f960a88dfac51312c01437770"} Nov 03 20:02:19.836195 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.836171 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.895443 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-istio-token\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-istio-token\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.895492 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-kube-api-access-m7zv2\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.895659 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5374374-e149-4721-beaa-b32f10b11ea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5374374-e149-4721-beaa-b32f10b11ea9-istio-podinfo\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.895789 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-envoy\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.895855 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-socket\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.896345 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5374374-e149-4721-beaa-b32f10b11ea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5374374-e149-4721-beaa-b32f10b11ea9-istiod-ca-cert\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.896433 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-credential-socket\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.896500 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-certs\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.897641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.896599 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-data\") pod \"f5374374-e149-4721-beaa-b32f10b11ea9\" (UID: \"f5374374-e149-4721-beaa-b32f10b11ea9\") " Nov 03 20:02:19.898741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.898464 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-data" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:19.899465 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.899251 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5374374-e149-4721-beaa-b32f10b11ea9-istiod-ca-cert" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:19.899772 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.899610 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-socket" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:19.900072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.899908 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-credential-socket" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:19.900231 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.900200 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-certs" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:19.912091 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.912051 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5374374-e149-4721-beaa-b32f10b11ea9-istio-podinfo" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:19.912448 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.912420 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-kube-api-access-m7zv2" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "kube-api-access-m7zv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:19.913741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.913714 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-envoy" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:19.914670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.914642 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-istio-token" pod "f5374374-e149-4721-beaa-b32f10b11ea9" (UID: "f5374374-e149-4721-beaa-b32f10b11ea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:19.998938 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.998902 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.998938 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.998940 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.999134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.998956 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7zv2\" (UniqueName: \"kubernetes.io/projected/f5374374-e149-4721-beaa-b32f10b11ea9-kube-api-access-m7zv2\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.999134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.998971 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5374374-e149-4721-beaa-b32f10b11ea9-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.999134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.998986 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.999134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.998999 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.999134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.999013 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5374374-e149-4721-beaa-b32f10b11ea9-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.999134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.999026 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:19.999134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:19.999040 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5374374-e149-4721-beaa-b32f10b11ea9-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:20.049171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.049090 4314 generic.go:356] "Generic (PLEG): container finished" podID="f5374374-e149-4721-beaa-b32f10b11ea9" containerID="1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547" exitCode=0 Nov 03 20:02:20.049171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.049138 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" Nov 03 20:02:20.049626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.049173 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" event={"ID":"f5374374-e149-4721-beaa-b32f10b11ea9","Type":"ContainerDied","Data":"1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547"} Nov 03 20:02:20.049626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.049231 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x" event={"ID":"f5374374-e149-4721-beaa-b32f10b11ea9","Type":"ContainerDied","Data":"29b1f02c548d74874c03007460c8cc027ac6e069a64dc5dec382a5ad084428dc"} Nov 03 20:02:20.049626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.049260 4314 scope.go:122] "RemoveContainer" containerID="1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547" Nov 03 20:02:20.068194 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.067867 4314 scope.go:122] "RemoveContainer" containerID="849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41" Nov 03 20:02:20.099640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.098531 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x"] Nov 03 20:02:20.102631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.100952 4314 scope.go:122] "RemoveContainer" containerID="1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547" Nov 03 20:02:20.102631 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:20.101342 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547\": container with ID starting with 1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547 not found: ID does not exist" containerID="1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547" Nov 03 20:02:20.102631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.101376 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547"} err="failed to get container status \"1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547\": rpc error: code = NotFound desc = could not find container \"1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547\": container with ID starting with 1bfc264f0b78c436de81e71dd77933404f4066e8ab4a587b10b348eca5f04547 not found: ID does not exist" Nov 03 20:02:20.102631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.101408 4314 scope.go:122] "RemoveContainer" containerID="849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41" Nov 03 20:02:20.102631 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:20.101845 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41\": container with ID starting with 849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41 not found: ID does not exist" containerID="849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41" Nov 03 20:02:20.102631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.101875 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41"} err="failed to get container status \"849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41\": rpc error: code = NotFound desc = could not find container \"849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41\": container with ID starting with 849cbb06f5213a080cf54610deb867b8bb9abb4e342d8fd0ffb6388b3fb4ce41 not found: ID does not exist" Nov 03 20:02:20.103831 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.103807 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2747a4-openshift-default-89d775bdf-j8d6x"] Nov 03 20:02:20.329631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.329595 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:20.412860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.412823 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e67e41-fb69-412b-aa31-03c50976ef72-kube-api-access-cfvpg\" (UniqueName: \"kubernetes.io/projected/d5e67e41-fb69-412b-aa31-03c50976ef72-kube-api-access-cfvpg\") pod \"d5e67e41-fb69-412b-aa31-03c50976ef72\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " Nov 03 20:02:20.413115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.413096 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-xks-ca\") pod \"d5e67e41-fb69-412b-aa31-03c50976ef72\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " Nov 03 20:02:20.413193 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.413166 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-openshift-ca\") pod \"d5e67e41-fb69-412b-aa31-03c50976ef72\" (UID: \"d5e67e41-fb69-412b-aa31-03c50976ef72\") " Nov 03 20:02:20.413725 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.413661 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-openshift-ca" pod "d5e67e41-fb69-412b-aa31-03c50976ef72" (UID: "d5e67e41-fb69-412b-aa31-03c50976ef72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:20.414011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.413989 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:20.417062 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.417039 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e67e41-fb69-412b-aa31-03c50976ef72-kube-api-access-cfvpg" pod "d5e67e41-fb69-412b-aa31-03c50976ef72" (UID: "d5e67e41-fb69-412b-aa31-03c50976ef72"). InnerVolumeSpecName "kube-api-access-cfvpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:20.417180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.417161 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-xks-ca" pod "d5e67e41-fb69-412b-aa31-03c50976ef72" (UID: "d5e67e41-fb69-412b-aa31-03c50976ef72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:20.433670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.433644 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" path="/var/lib/kubelet/pods/f5374374-e149-4721-beaa-b32f10b11ea9/volumes" Nov 03 20:02:20.516234 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.516200 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfvpg\" (UniqueName: \"kubernetes.io/projected/d5e67e41-fb69-412b-aa31-03c50976ef72-kube-api-access-cfvpg\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:20.516234 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:20.516230 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e67e41-fb69-412b-aa31-03c50976ef72-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:21.128650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.128606 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" Nov 03 20:02:21.131105 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.128954 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg" event={"ID":"d5e67e41-fb69-412b-aa31-03c50976ef72","Type":"ContainerDied","Data":"8b32f7ace3417f6710f1d21f8405c9047201c9b8ea21f19f84cfab0625abd3d8"} Nov 03 20:02:21.131269 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.131138 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b32f7ace3417f6710f1d21f8405c9047201c9b8ea21f19f84cfab0625abd3d8" Nov 03 20:02:21.326615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.316559 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb"] Nov 03 20:02:21.381139 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.371620 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx"] Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.468683 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp"] Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469574 4314 cpu_manager.go:414] "containerMap: removing container" podUID="d5e67e41-fb69-412b-aa31-03c50976ef72" containerName="revoke-keys" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469608 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e67e41-fb69-412b-aa31-03c50976ef72" containerName="revoke-keys" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469636 4314 cpu_manager.go:414] "containerMap: removing container" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469642 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469667 4314 cpu_manager.go:414] "containerMap: removing container" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469673 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469804 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e67e41-fb69-412b-aa31-03c50976ef72" containerName="revoke-keys" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469818 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" Nov 03 20:02:21.477634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.469828 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5374374-e149-4721-beaa-b32f10b11ea9" containerName="istio-proxy" Nov 03 20:02:21.483635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.479969 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp"] Nov 03 20:02:21.483635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.480090 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.516404 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.516371 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:02:21.516624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.516447 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:21.559183 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.559150 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxkrn\" (UniqueName: \"kubernetes.io/projected/ca102aed-069f-48db-b3ca-05f91cc032d1-kube-api-access-gxkrn\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.559391 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.559196 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.559455 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.559405 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.661234 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.661150 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.661422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.661377 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxkrn\" (UniqueName: \"kubernetes.io/projected/ca102aed-069f-48db-b3ca-05f91cc032d1-kube-api-access-gxkrn\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.661498 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.661423 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.662278 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.662249 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.663941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.663915 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.669745 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.669719 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxkrn\" (UniqueName: \"kubernetes.io/projected/ca102aed-069f-48db-b3ca-05f91cc032d1-kube-api-access-gxkrn\") pod \"maas-api-revoke-sub-b9abd7e0-zj2mp\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:21.816537 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:21.816506 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:22.005641 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:22.002482 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:02:22.130217 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:02:22.130182 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca102aed_069f_48db_b3ca_05f91cc032d1.slice/crio-ea0ed632d15c27cf09dc2bf8140b1ab4b905a292999e2ae40dd926bd82e180be WatchSource:0}: Error finding container ea0ed632d15c27cf09dc2bf8140b1ab4b905a292999e2ae40dd926bd82e180be: Status 404 returned error can't find the container with id ea0ed632d15c27cf09dc2bf8140b1ab4b905a292999e2ae40dd926bd82e180be Nov 03 20:02:22.131428 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:22.131401 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp"] Nov 03 20:02:22.173549 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:22.173521 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" event={"ID":"ca102aed-069f-48db-b3ca-05f91cc032d1","Type":"ContainerStarted","Data":"ea0ed632d15c27cf09dc2bf8140b1ab4b905a292999e2ae40dd926bd82e180be"} Nov 03 20:02:23.185501 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:23.185458 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" event={"ID":"ca102aed-069f-48db-b3ca-05f91cc032d1","Type":"ContainerStarted","Data":"4d8be0e97654dc100bc057cef35828c46a1217a76cb7b72a46b89c0b0a778461"} Nov 03 20:02:23.207656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:23.207605 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" podStartSLOduration=2.207591095 podStartE2EDuration="2.207591095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:23.201349031 +0000 UTC m=+1850.906188370" watchObservedRunningTime="2026-11-03 20:02:23.207591095 +0000 UTC m=+1850.912430432" Nov 03 20:02:24.198465 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:24.198427 4314 generic.go:356] "Generic (PLEG): container finished" podID="ca102aed-069f-48db-b3ca-05f91cc032d1" containerID="4d8be0e97654dc100bc057cef35828c46a1217a76cb7b72a46b89c0b0a778461" exitCode=0 Nov 03 20:02:24.198837 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:24.198522 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" event={"ID":"ca102aed-069f-48db-b3ca-05f91cc032d1","Type":"ContainerDied","Data":"4d8be0e97654dc100bc057cef35828c46a1217a76cb7b72a46b89c0b0a778461"} Nov 03 20:02:25.476980 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.476905 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:25.533282 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.533226 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-openshift-ca\") pod \"ca102aed-069f-48db-b3ca-05f91cc032d1\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " Nov 03 20:02:25.533617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.533562 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca102aed-069f-48db-b3ca-05f91cc032d1-kube-api-access-gxkrn\" (UniqueName: \"kubernetes.io/projected/ca102aed-069f-48db-b3ca-05f91cc032d1-kube-api-access-gxkrn\") pod \"ca102aed-069f-48db-b3ca-05f91cc032d1\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " Nov 03 20:02:25.533785 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.533761 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-xks-ca\") pod \"ca102aed-069f-48db-b3ca-05f91cc032d1\" (UID: \"ca102aed-069f-48db-b3ca-05f91cc032d1\") " Nov 03 20:02:25.535978 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.535944 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-openshift-ca" pod "ca102aed-069f-48db-b3ca-05f91cc032d1" (UID: "ca102aed-069f-48db-b3ca-05f91cc032d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:25.537776 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.537743 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-xks-ca" pod "ca102aed-069f-48db-b3ca-05f91cc032d1" (UID: "ca102aed-069f-48db-b3ca-05f91cc032d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:25.538474 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.538445 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca102aed-069f-48db-b3ca-05f91cc032d1-kube-api-access-gxkrn" pod "ca102aed-069f-48db-b3ca-05f91cc032d1" (UID: "ca102aed-069f-48db-b3ca-05f91cc032d1"). InnerVolumeSpecName "kube-api-access-gxkrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:25.636174 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.636140 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:25.636174 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.636169 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca102aed-069f-48db-b3ca-05f91cc032d1-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:25.636174 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:25.636180 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxkrn\" (UniqueName: \"kubernetes.io/projected/ca102aed-069f-48db-b3ca-05f91cc032d1-kube-api-access-gxkrn\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:26.216082 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:26.216050 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" Nov 03 20:02:26.216271 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:26.216080 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp" event={"ID":"ca102aed-069f-48db-b3ca-05f91cc032d1","Type":"ContainerDied","Data":"ea0ed632d15c27cf09dc2bf8140b1ab4b905a292999e2ae40dd926bd82e180be"} Nov 03 20:02:26.216271 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:26.216129 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0ed632d15c27cf09dc2bf8140b1ab4b905a292999e2ae40dd926bd82e180be" Nov 03 20:02:26.957604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:26.943083 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c"] Nov 03 20:02:26.987718 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:26.979547 4314 cpu_manager.go:414] "containerMap: removing container" podUID="ca102aed-069f-48db-b3ca-05f91cc032d1" containerName="revoke-keys" Nov 03 20:02:26.988140 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:26.988116 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca102aed-069f-48db-b3ca-05f91cc032d1" containerName="revoke-keys" Nov 03 20:02:27.003602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:26.988645 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca102aed-069f-48db-b3ca-05f91cc032d1" containerName="revoke-keys" Nov 03 20:02:27.216658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.189085 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.224636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.211965 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgrd9\" (UniqueName: \"kubernetes.io/projected/1eabd202-7812-434f-b7d7-9af6b1532f42-kube-api-access-dgrd9\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.224636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.220745 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.224636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.220892 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.224636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.220948 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eabd202-7812-434f-b7d7-9af6b1532f42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.224636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.215661 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c"] Nov 03 20:02:27.267129 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.267068 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-3ff278-serving-cert\"" Nov 03 20:02:27.341628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.328791 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.341628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.328953 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.341628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.329019 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eabd202-7812-434f-b7d7-9af6b1532f42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.341628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.329402 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgrd9\" (UniqueName: \"kubernetes.io/projected/1eabd202-7812-434f-b7d7-9af6b1532f42-kube-api-access-dgrd9\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.341628 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:27.331639 4314 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-serving-cert: secret "maas-api-e2e-shared-a-w0-3ff278-serving-cert" not found Nov 03 20:02:27.341628 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:27.331724 4314 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls podName:1eabd202-7812-434f-b7d7-9af6b1532f42 nodeName:}" failed. No retries permitted until 2026-11-03 20:02:27.83170398 +0000 UTC m=+1855.536543303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls") pod "maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" (UID: "1eabd202-7812-434f-b7d7-9af6b1532f42") : secret "maas-api-e2e-shared-a-w0-3ff278-serving-cert" not found Nov 03 20:02:27.341628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.332622 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eabd202-7812-434f-b7d7-9af6b1532f42-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.365632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.358200 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.391631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.390723 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgrd9\" (UniqueName: \"kubernetes.io/projected/1eabd202-7812-434f-b7d7-9af6b1532f42-kube-api-access-dgrd9\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.851680 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.841341 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.896606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.896259 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:27.926605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:27.925938 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:28.528888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:28.527489 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c"] Nov 03 20:02:28.543785 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:02:28.542274 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eabd202_7812_434f_b7d7_9af6b1532f42.slice/crio-f068b3f3ec869bbd47e27f9bd285576e80ebb822d9786c1912087ec022d9725b WatchSource:0}: Error finding container f068b3f3ec869bbd47e27f9bd285576e80ebb822d9786c1912087ec022d9725b: Status 404 returned error can't find the container with id f068b3f3ec869bbd47e27f9bd285576e80ebb822d9786c1912087ec022d9725b Nov 03 20:02:29.456053 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:29.455968 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" event={"ID":"1eabd202-7812-434f-b7d7-9af6b1532f42","Type":"ContainerStarted","Data":"8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77"} Nov 03 20:02:29.456053 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:29.456026 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" event={"ID":"1eabd202-7812-434f-b7d7-9af6b1532f42","Type":"ContainerStarted","Data":"f068b3f3ec869bbd47e27f9bd285576e80ebb822d9786c1912087ec022d9725b"} Nov 03 20:02:29.456262 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:29.456207 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:29.477942 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:29.477893 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" podStartSLOduration=2.844197621 podStartE2EDuration="3.477880058s" totalImagesPullingTime="633.682437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:29.472025331 +0000 UTC m=+1857.176864672" watchObservedRunningTime="2026-11-03 20:02:29.477880058 +0000 UTC m=+1857.182719397" Nov 03 20:02:35.473165 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.473133 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:02:35.733991 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.733908 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62"] Nov 03 20:02:35.740289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.740263 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.744344 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.744321 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:02:35.744537 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.744339 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:02:35.746378 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.746355 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62"] Nov 03 20:02:35.774074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.767939 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802abf8e-180b-4020-af28-64a5010ef933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.774074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.768063 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802abf8e-180b-4020-af28-64a5010ef933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.774074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.768173 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msx24\" (UniqueName: \"kubernetes.io/projected/802abf8e-180b-4020-af28-64a5010ef933-kube-api-access-msx24\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.869518 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.869479 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802abf8e-180b-4020-af28-64a5010ef933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.869718 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.869679 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msx24\" (UniqueName: \"kubernetes.io/projected/802abf8e-180b-4020-af28-64a5010ef933-kube-api-access-msx24\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.869949 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.869926 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802abf8e-180b-4020-af28-64a5010ef933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.870244 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.870218 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802abf8e-180b-4020-af28-64a5010ef933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.872284 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.872255 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802abf8e-180b-4020-af28-64a5010ef933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:35.878459 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:35.878437 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msx24\" (UniqueName: \"kubernetes.io/projected/802abf8e-180b-4020-af28-64a5010ef933-kube-api-access-msx24\") pod \"maas-api-revoke-sub-3f210049-tqp62\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:36.074027 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.073990 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:36.435508 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.435425 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" containerID="cri-o://dd0519ace3664cc27ecbe80dd4c4b90dfe3075c1208123526a22985a43b40812" gracePeriod=45 Nov 03 20:02:36.453480 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:02:36.453454 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802abf8e_180b_4020_af28_64a5010ef933.slice/crio-644953bb2f1a9e5a42bb0a0eb2bae5e9a1fcba5bb80c9b9409d979540595ed9f WatchSource:0}: Error finding container 644953bb2f1a9e5a42bb0a0eb2bae5e9a1fcba5bb80c9b9409d979540595ed9f: Status 404 returned error can't find the container with id 644953bb2f1a9e5a42bb0a0eb2bae5e9a1fcba5bb80c9b9409d979540595ed9f Nov 03 20:02:36.456543 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.456514 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62"] Nov 03 20:02:36.507533 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.507486 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" containerID="cri-o://96f2a89509b3780b0c37ae0457e5bb32c71f5bb4f7a176c208878bc14e55af3a" gracePeriod=45 Nov 03 20:02:36.673270 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.673124 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused" Nov 03 20:02:36.818102 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.814609 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 20:02:36.856346 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.856312 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890399 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-home\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-home\") pod \"cfa79738-f4df-4b25-9319-ab36d813b7b9\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890469 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-dshm\") pod \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890528 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tmp-dir\") pod \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890600 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa79738-f4df-4b25-9319-ab36d813b7b9-kube-api-access-rcfvc\" (UniqueName: \"kubernetes.io/projected/cfa79738-f4df-4b25-9319-ab36d813b7b9-kube-api-access-rcfvc\") pod \"cfa79738-f4df-4b25-9319-ab36d813b7b9\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890650 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa79738-f4df-4b25-9319-ab36d813b7b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa79738-f4df-4b25-9319-ab36d813b7b9-tls-certs\") pod \"cfa79738-f4df-4b25-9319-ab36d813b7b9\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890685 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tls-certs\") pod \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890740 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-model-cache\") pod \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890781 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-kube-api-access-d6zgb\" (UniqueName: \"kubernetes.io/projected/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-kube-api-access-d6zgb\") pod \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890839 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-model-cache\") pod \"cfa79738-f4df-4b25-9319-ab36d813b7b9\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.890961 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-dshm\") pod \"cfa79738-f4df-4b25-9319-ab36d813b7b9\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.891024 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-home\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-home\") pod \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\" (UID: \"a318f53f-7b2c-4b68-96e9-51fcd403c2d6\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.891104 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-tmp-dir\") pod \"cfa79738-f4df-4b25-9319-ab36d813b7b9\" (UID: \"cfa79738-f4df-4b25-9319-ab36d813b7b9\") " Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.892715 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-tmp-dir" pod "cfa79738-f4df-4b25-9319-ab36d813b7b9" (UID: "cfa79738-f4df-4b25-9319-ab36d813b7b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.892919 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-model-cache" pod "a318f53f-7b2c-4b68-96e9-51fcd403c2d6" (UID: "a318f53f-7b2c-4b68-96e9-51fcd403c2d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.895359 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-kube-api-access-d6zgb" pod "a318f53f-7b2c-4b68-96e9-51fcd403c2d6" (UID: "a318f53f-7b2c-4b68-96e9-51fcd403c2d6"). InnerVolumeSpecName "kube-api-access-d6zgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:36.896953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.895719 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-model-cache" pod "cfa79738-f4df-4b25-9319-ab36d813b7b9" (UID: "cfa79738-f4df-4b25-9319-ab36d813b7b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.898125 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.897568 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-dshm" pod "cfa79738-f4df-4b25-9319-ab36d813b7b9" (UID: "cfa79738-f4df-4b25-9319-ab36d813b7b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.898125 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.897791 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-home" pod "a318f53f-7b2c-4b68-96e9-51fcd403c2d6" (UID: "a318f53f-7b2c-4b68-96e9-51fcd403c2d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.898125 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.897913 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tmp-dir" pod "a318f53f-7b2c-4b68-96e9-51fcd403c2d6" (UID: "a318f53f-7b2c-4b68-96e9-51fcd403c2d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.898125 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.898025 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-home" pod "cfa79738-f4df-4b25-9319-ab36d813b7b9" (UID: "cfa79738-f4df-4b25-9319-ab36d813b7b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.904166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.899717 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-dshm" pod "a318f53f-7b2c-4b68-96e9-51fcd403c2d6" (UID: "a318f53f-7b2c-4b68-96e9-51fcd403c2d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:36.904166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.901804 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa79738-f4df-4b25-9319-ab36d813b7b9-kube-api-access-rcfvc" pod "cfa79738-f4df-4b25-9319-ab36d813b7b9" (UID: "cfa79738-f4df-4b25-9319-ab36d813b7b9"). InnerVolumeSpecName "kube-api-access-rcfvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:36.904166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.903955 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tls-certs" pod "a318f53f-7b2c-4b68-96e9-51fcd403c2d6" (UID: "a318f53f-7b2c-4b68-96e9-51fcd403c2d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:36.904166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.904026 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa79738-f4df-4b25-9319-ab36d813b7b9-tls-certs" pod "cfa79738-f4df-4b25-9319-ab36d813b7b9" (UID: "cfa79738-f4df-4b25-9319-ab36d813b7b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:36.993782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993740 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993780 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfvc\" (UniqueName: \"kubernetes.io/projected/cfa79738-f4df-4b25-9319-ab36d813b7b9-kube-api-access-rcfvc\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993795 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfa79738-f4df-4b25-9319-ab36d813b7b9-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993806 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993818 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993832 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zgb\" (UniqueName: \"kubernetes.io/projected/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-kube-api-access-d6zgb\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993844 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993860 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993872 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993882 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993895 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfa79738-f4df-4b25-9319-ab36d813b7b9-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:36.993986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:36.993906 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a318f53f-7b2c-4b68-96e9-51fcd403c2d6-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:37.182692 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.180450 4314 scope.go:122] "RemoveContainer" containerID="dd0519ace3664cc27ecbe80dd4c4b90dfe3075c1208123526a22985a43b40812" Nov 03 20:02:37.219872 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.219848 4314 scope.go:122] "RemoveContainer" containerID="96f2a89509b3780b0c37ae0457e5bb32c71f5bb4f7a176c208878bc14e55af3a" Nov 03 20:02:37.237952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.237932 4314 scope.go:122] "RemoveContainer" containerID="7f7e994fbb42714ed8b8df2e7bf23646e5b818981af4d1244914e98a30f9a781" Nov 03 20:02:37.549206 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549117 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" event={"ID":"802abf8e-180b-4020-af28-64a5010ef933","Type":"ContainerStarted","Data":"2d5e8fcbd5d347cf1ab599541acb4bd9b3f265f2fd0a8e46d5c27a2e6dcba048"} Nov 03 20:02:37.549206 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549203 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" event={"ID":"802abf8e-180b-4020-af28-64a5010ef933","Type":"ContainerStarted","Data":"644953bb2f1a9e5a42bb0a0eb2bae5e9a1fcba5bb80c9b9409d979540595ed9f"} Nov 03 20:02:37.549633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549234 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" event={"ID":"a318f53f-7b2c-4b68-96e9-51fcd403c2d6","Type":"ContainerDied","Data":"96f2a89509b3780b0c37ae0457e5bb32c71f5bb4f7a176c208878bc14e55af3a"} Nov 03 20:02:37.549633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549254 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" event={"ID":"a318f53f-7b2c-4b68-96e9-51fcd403c2d6","Type":"ContainerDied","Data":"ef1a3fc899337af59208f1dc007fee762cb6985236aa173ee2eabc8254d9ccb7"} Nov 03 20:02:37.549633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549269 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" event={"ID":"cfa79738-f4df-4b25-9319-ab36d813b7b9","Type":"ContainerDied","Data":"dd0519ace3664cc27ecbe80dd4c4b90dfe3075c1208123526a22985a43b40812"} Nov 03 20:02:37.549633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549281 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" event={"ID":"cfa79738-f4df-4b25-9319-ab36d813b7b9","Type":"ContainerDied","Data":"09ea4cc8018472fe8985371b66c8b1a54eebf54e40861c6d925c19a5bde83b6b"} Nov 03 20:02:37.549633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549336 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx" Nov 03 20:02:37.549633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.549336 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb" Nov 03 20:02:37.569174 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.569123 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" podStartSLOduration=2.569109958 podStartE2EDuration="2.569109958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:37.563897312 +0000 UTC m=+1865.268736676" watchObservedRunningTime="2026-11-03 20:02:37.569109958 +0000 UTC m=+1865.273949324" Nov 03 20:02:37.596544 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.596511 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx"] Nov 03 20:02:37.601415 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.601390 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-102379/premium-w6-102379-kserve-5678c74646-pnrpx"] Nov 03 20:02:37.619347 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.619318 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb"] Nov 03 20:02:37.623197 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:37.623174 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-w6-102379-kserve-78d65d6fb5-cxxnb"] Nov 03 20:02:38.431703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:38.431673 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" path="/var/lib/kubelet/pods/a318f53f-7b2c-4b68-96e9-51fcd403c2d6/volumes" Nov 03 20:02:38.432158 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:38.432143 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" path="/var/lib/kubelet/pods/cfa79738-f4df-4b25-9319-ab36d813b7b9/volumes" Nov 03 20:02:38.617082 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:38.617045 4314 generic.go:356] "Generic (PLEG): container finished" podID="802abf8e-180b-4020-af28-64a5010ef933" containerID="2d5e8fcbd5d347cf1ab599541acb4bd9b3f265f2fd0a8e46d5c27a2e6dcba048" exitCode=0 Nov 03 20:02:38.617715 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:38.617128 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" event={"ID":"802abf8e-180b-4020-af28-64a5010ef933","Type":"ContainerDied","Data":"2d5e8fcbd5d347cf1ab599541acb4bd9b3f265f2fd0a8e46d5c27a2e6dcba048"} Nov 03 20:02:40.009562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.009179 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:40.066295 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.066256 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/802abf8e-180b-4020-af28-64a5010ef933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802abf8e-180b-4020-af28-64a5010ef933-maas-api-xks-ca\") pod \"802abf8e-180b-4020-af28-64a5010ef933\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " Nov 03 20:02:40.067035 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.067011 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802abf8e-180b-4020-af28-64a5010ef933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802abf8e-180b-4020-af28-64a5010ef933-maas-api-openshift-ca\") pod \"802abf8e-180b-4020-af28-64a5010ef933\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " Nov 03 20:02:40.071235 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.071211 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802abf8e-180b-4020-af28-64a5010ef933-kube-api-access-msx24\" (UniqueName: \"kubernetes.io/projected/802abf8e-180b-4020-af28-64a5010ef933-kube-api-access-msx24\") pod \"802abf8e-180b-4020-af28-64a5010ef933\" (UID: \"802abf8e-180b-4020-af28-64a5010ef933\") " Nov 03 20:02:40.071613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.071116 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802abf8e-180b-4020-af28-64a5010ef933-maas-api-openshift-ca" pod "802abf8e-180b-4020-af28-64a5010ef933" (UID: "802abf8e-180b-4020-af28-64a5010ef933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:40.079624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.074538 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802abf8e-180b-4020-af28-64a5010ef933-kube-api-access-msx24" pod "802abf8e-180b-4020-af28-64a5010ef933" (UID: "802abf8e-180b-4020-af28-64a5010ef933"). InnerVolumeSpecName "kube-api-access-msx24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:40.080268 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.079892 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/802abf8e-180b-4020-af28-64a5010ef933-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:40.101603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.090710 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802abf8e-180b-4020-af28-64a5010ef933-maas-api-xks-ca" pod "802abf8e-180b-4020-af28-64a5010ef933" (UID: "802abf8e-180b-4020-af28-64a5010ef933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:40.186655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.181483 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/802abf8e-180b-4020-af28-64a5010ef933-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:40.186655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.181520 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msx24\" (UniqueName: \"kubernetes.io/projected/802abf8e-180b-4020-af28-64a5010ef933-kube-api-access-msx24\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:40.433380 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.433344 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc"] Nov 03 20:02:40.433596 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.433407 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj"] Nov 03 20:02:40.433687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.433621 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" containerID="cri-o://8e05e8c3ddecd72dadafacf4ac13b1387aaf034591a5d95ed53c30c40a053df9" gracePeriod=30 Nov 03 20:02:40.433753 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.433689 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" containerID="cri-o://1c2844bdafa19e6f2c0dbcdcb434d9ed0d49f55cb49586281f346cf8209ee7c6" gracePeriod=30 Nov 03 20:02:40.638397 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.638363 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" event={"ID":"802abf8e-180b-4020-af28-64a5010ef933","Type":"ContainerDied","Data":"644953bb2f1a9e5a42bb0a0eb2bae5e9a1fcba5bb80c9b9409d979540595ed9f"} Nov 03 20:02:40.638397 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.638399 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62" Nov 03 20:02:40.638689 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:40.638414 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644953bb2f1a9e5a42bb0a0eb2bae5e9a1fcba5bb80c9b9409d979540595ed9f" Nov 03 20:02:42.996441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:42.991259 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn"] Nov 03 20:02:42.996441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:42.991569 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" containerID="cri-o://aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931" gracePeriod=30 Nov 03 20:02:45.711976 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.711838 4314 generic.go:356] "Generic (PLEG): container finished" podID="202ad091-28ed-41c2-aeae-d60761a4d034" containerID="8e05e8c3ddecd72dadafacf4ac13b1387aaf034591a5d95ed53c30c40a053df9" exitCode=0 Nov 03 20:02:45.712806 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.712713 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" event={"ID":"202ad091-28ed-41c2-aeae-d60761a4d034","Type":"ContainerDied","Data":"8e05e8c3ddecd72dadafacf4ac13b1387aaf034591a5d95ed53c30c40a053df9"} Nov 03 20:02:45.717384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.717354 4314 generic.go:356] "Generic (PLEG): container finished" podID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerID="1c2844bdafa19e6f2c0dbcdcb434d9ed0d49f55cb49586281f346cf8209ee7c6" exitCode=0 Nov 03 20:02:45.717675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.717466 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" event={"ID":"b7e969e0-d5d8-42d3-a243-fdba3ec8534f","Type":"ContainerDied","Data":"1c2844bdafa19e6f2c0dbcdcb434d9ed0d49f55cb49586281f346cf8209ee7c6"} Nov 03 20:02:45.845069 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.845040 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 20:02:45.889997 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.889740 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 20:02:45.923649 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923611 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-socket\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.923834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923714 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-credential-socket\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.923834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923795 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-kube-api-access-p7nm4\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-kube-api-access-p7nm4\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.923962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923842 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-envoy\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.923962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923892 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-data\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.923962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923936 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-credential-socket\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.923962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923938 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-socket" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.924171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.923982 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-kube-api-access-hn4k4\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-kube-api-access-hn4k4\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.924171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924072 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-podinfo\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.924171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924117 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-istio-token\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-istio-token\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.924322 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924180 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/202ad091-28ed-41c2-aeae-d60761a4d034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202ad091-28ed-41c2-aeae-d60761a4d034-istio-podinfo\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.924322 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924232 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-data\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.924322 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924285 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-envoy\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.924322 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924309 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istiod-ca-cert\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.924540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924335 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-certs\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.924540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924363 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-socket\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.924540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924481 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-token\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-token\") pod \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\" (UID: \"b7e969e0-d5d8-42d3-a243-fdba3ec8534f\") " Nov 03 20:02:45.924540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924498 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202ad091-28ed-41c2-aeae-d60761a4d034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202ad091-28ed-41c2-aeae-d60761a4d034-istiod-ca-cert\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.924785 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.924562 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-certs\") pod \"202ad091-28ed-41c2-aeae-d60761a4d034\" (UID: \"202ad091-28ed-41c2-aeae-d60761a4d034\") " Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.925050 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-data" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.925287 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-certs" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.925304 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-credential-socket" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.925547 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-certs" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926044 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926067 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926082 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926097 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:45.926402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926113 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:45.927904 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926466 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-envoy" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.927904 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926642 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-data" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.927904 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.926904 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-istio-token" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:45.927904 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.927027 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istiod-ca-cert" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:45.927904 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.927344 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202ad091-28ed-41c2-aeae-d60761a4d034-istiod-ca-cert" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:45.927904 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.927520 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-socket" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.927904 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.927564 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-kube-api-access-hn4k4" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "kube-api-access-hn4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:45.928417 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.928395 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-credential-socket" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.928530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.928459 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/202ad091-28ed-41c2-aeae-d60761a4d034-istio-podinfo" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:45.928674 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.928657 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-podinfo" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:45.929863 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.929841 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-envoy" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:45.929945 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.929883 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-kube-api-access-p7nm4" pod "202ad091-28ed-41c2-aeae-d60761a4d034" (UID: "202ad091-28ed-41c2-aeae-d60761a4d034"). InnerVolumeSpecName "kube-api-access-p7nm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:45.931389 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:45.931367 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-token" pod "b7e969e0-d5d8-42d3-a243-fdba3ec8534f" (UID: "b7e969e0-d5d8-42d3-a243-fdba3ec8534f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:46.032374 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032318 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/202ad091-28ed-41c2-aeae-d60761a4d034-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032374 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032348 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032374 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032357 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032374 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032369 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032387 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032401 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032410 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/202ad091-28ed-41c2-aeae-d60761a4d034-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032430 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/202ad091-28ed-41c2-aeae-d60761a4d034-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032445 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7nm4\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-kube-api-access-p7nm4\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032458 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032471 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4k4\" (UniqueName: \"kubernetes.io/projected/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-kube-api-access-hn4k4\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032482 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7e969e0-d5d8-42d3-a243-fdba3ec8534f-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.032762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.032496 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/202ad091-28ed-41c2-aeae-d60761a4d034-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:46.727471 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.727388 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" Nov 03 20:02:46.727471 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.727413 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc" event={"ID":"b7e969e0-d5d8-42d3-a243-fdba3ec8534f","Type":"ContainerDied","Data":"52ef1d6df3dca96232e9bed0e5e1dd0770a7d3aaed984d5fb146d8e780b88264"} Nov 03 20:02:46.727971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.727474 4314 scope.go:122] "RemoveContainer" containerID="1c2844bdafa19e6f2c0dbcdcb434d9ed0d49f55cb49586281f346cf8209ee7c6" Nov 03 20:02:46.729326 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.729302 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" event={"ID":"202ad091-28ed-41c2-aeae-d60761a4d034","Type":"ContainerDied","Data":"19281904efae9196619d4cb6bb73d6b568a0e8cd8c13646c2ee0698b0f2ae302"} Nov 03 20:02:46.729459 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.729344 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj" Nov 03 20:02:46.740093 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.740072 4314 scope.go:122] "RemoveContainer" containerID="8e05e8c3ddecd72dadafacf4ac13b1387aaf034591a5d95ed53c30c40a053df9" Nov 03 20:02:46.753124 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.753102 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc"] Nov 03 20:02:46.757472 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.757450 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-76cxc"] Nov 03 20:02:46.769960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.769935 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj"] Nov 03 20:02:46.774366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:46.774335 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-57ktj"] Nov 03 20:02:48.433748 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.433668 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" path="/var/lib/kubelet/pods/202ad091-28ed-41c2-aeae-d60761a4d034/volumes" Nov 03 20:02:48.434409 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.434389 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" path="/var/lib/kubelet/pods/b7e969e0-d5d8-42d3-a243-fdba3ec8534f/volumes" Nov 03 20:02:48.613826 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.613076 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 20:02:48.688923 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.688832 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-kube-api-access-5q69j\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-kube-api-access-5q69j\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.688923 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.688894 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-token\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-token\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.688939 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-credential-socket\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689015 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-data\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689108 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-socket\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689141 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-podinfo\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689361 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689191 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-certs\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689361 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689257 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-envoy\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689361 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689321 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istiod-ca-cert\") pod \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\" (UID: \"a0209b03-fcc6-4a8c-97b8-edefc36eb120\") " Nov 03 20:02:48.689507 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689462 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-data" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689689 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-credential-socket" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.689799 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-certs" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.690125 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istiod-ca-cert" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.690277 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.690299 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.690313 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.690327 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.692445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.690367 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-socket" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:48.693454 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.693419 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-envoy" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:48.693671 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.693636 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-podinfo" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:48.693931 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.693903 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-kube-api-access-5q69j" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "kube-api-access-5q69j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:48.707103 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.705759 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-token" pod "a0209b03-fcc6-4a8c-97b8-edefc36eb120" (UID: "a0209b03-fcc6-4a8c-97b8-edefc36eb120"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:48.766715 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.766438 4314 generic.go:356] "Generic (PLEG): container finished" podID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerID="aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931" exitCode=0 Nov 03 20:02:48.766715 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.766625 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" event={"ID":"a0209b03-fcc6-4a8c-97b8-edefc36eb120","Type":"ContainerDied","Data":"aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931"} Nov 03 20:02:48.766715 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.766679 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" event={"ID":"a0209b03-fcc6-4a8c-97b8-edefc36eb120","Type":"ContainerDied","Data":"0161cf20dd399a93ed7169c80b11262a37447602036550815ce632c6c55f8c51"} Nov 03 20:02:48.767041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.766736 4314 scope.go:122] "RemoveContainer" containerID="aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931" Nov 03 20:02:48.767041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.766903 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn" Nov 03 20:02:48.785415 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.785395 4314 scope.go:122] "RemoveContainer" containerID="aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931" Nov 03 20:02:48.785760 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:48.785733 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931\": container with ID starting with aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931 not found: ID does not exist" containerID="aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931" Nov 03 20:02:48.785828 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.785778 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931"} err="failed to get container status \"aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931\": rpc error: code = NotFound desc = could not find container \"aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931\": container with ID starting with aae8bb2e069bdbf0831b812a52398863b0a9cd766770924c1e6e4895ab286931 not found: ID does not exist" Nov 03 20:02:48.792266 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.792146 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q69j\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-kube-api-access-5q69j\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.792266 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.792174 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.792266 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.792186 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.792266 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.792195 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.792266 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.792207 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0209b03-fcc6-4a8c-97b8-edefc36eb120-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:48.799347 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.799322 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn"] Nov 03 20:02:48.803871 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:48.803850 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-nssxn"] Nov 03 20:02:49.233418 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.229563 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr"] Nov 03 20:02:49.233418 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.229808 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" podUID="b7b22c85-eb65-412d-8cc8-48dd9e92aecb" containerName="maas-api" containerID="cri-o://b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80" gracePeriod=30 Nov 03 20:02:49.660277 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.656977 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 20:02:49.851656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.844187 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-openshift-service-ca\") pod \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " Nov 03 20:02:49.851656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.844238 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-maas-api-tls\") pod \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " Nov 03 20:02:49.851656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.844279 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-kube-api-access-zcs59\" (UniqueName: \"kubernetes.io/projected/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-kube-api-access-zcs59\") pod \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " Nov 03 20:02:49.851656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.844310 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-metrics-certs\") pod \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\" (UID: \"b7b22c85-eb65-412d-8cc8-48dd9e92aecb\") " Nov 03 20:02:49.893638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.888924 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-openshift-service-ca" pod "b7b22c85-eb65-412d-8cc8-48dd9e92aecb" (UID: "b7b22c85-eb65-412d-8cc8-48dd9e92aecb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:49.919605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.917909 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:02:49.960281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.959816 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:49.960281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.960030 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-kube-api-access-zcs59" pod "b7b22c85-eb65-412d-8cc8-48dd9e92aecb" (UID: "b7b22c85-eb65-412d-8cc8-48dd9e92aecb"). InnerVolumeSpecName "kube-api-access-zcs59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:49.974823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.974783 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-metrics-certs" pod "b7b22c85-eb65-412d-8cc8-48dd9e92aecb" (UID: "b7b22c85-eb65-412d-8cc8-48dd9e92aecb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:49.975553 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.975526 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-maas-api-tls" pod "b7b22c85-eb65-412d-8cc8-48dd9e92aecb" (UID: "b7b22c85-eb65-412d-8cc8-48dd9e92aecb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:49.983792 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.981192 4314 generic.go:356] "Generic (PLEG): container finished" podID="b7b22c85-eb65-412d-8cc8-48dd9e92aecb" containerID="b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80" exitCode=0 Nov 03 20:02:49.989519 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.983975 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" event={"ID":"b7b22c85-eb65-412d-8cc8-48dd9e92aecb","Type":"ContainerDied","Data":"b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80"} Nov 03 20:02:49.989746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.989712 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" event={"ID":"b7b22c85-eb65-412d-8cc8-48dd9e92aecb","Type":"ContainerDied","Data":"a88030d833700933dc1a86ea5242805465a37daff4967dd972026666c044b3cb"} Nov 03 20:02:49.989899 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.989884 4314 scope.go:122] "RemoveContainer" containerID="b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80" Nov 03 20:02:49.990118 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.984073 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr" Nov 03 20:02:49.993844 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.993237 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts"] Nov 03 20:02:49.993844 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:49.993488 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" containerID="cri-o://3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301" gracePeriod=30 Nov 03 20:02:50.021906 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.021816 4314 scope.go:122] "RemoveContainer" containerID="b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80" Nov 03 20:02:50.022378 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:50.022344 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80\": container with ID starting with b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80 not found: ID does not exist" containerID="b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80" Nov 03 20:02:50.022483 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.022387 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80"} err="failed to get container status \"b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80\": rpc error: code = NotFound desc = could not find container \"b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80\": container with ID starting with b6d9ef3d72d5c4b49fa2b7ef90d2eeca496fb7cf80aa0ebbebe74dc9c322bc80 not found: ID does not exist" Nov 03 20:02:50.055984 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.055944 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr"] Nov 03 20:02:50.078999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.076869 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:50.078999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.076902 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcs59\" (UniqueName: \"kubernetes.io/projected/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-kube-api-access-zcs59\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:50.078999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.076916 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b22c85-eb65-412d-8cc8-48dd9e92aecb-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:50.090372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.090341 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a99fad-5975cc4799-92qcr"] Nov 03 20:02:50.430897 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.430866 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" path="/var/lib/kubelet/pods/a0209b03-fcc6-4a8c-97b8-edefc36eb120/volumes" Nov 03 20:02:50.431425 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:50.431409 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b22c85-eb65-412d-8cc8-48dd9e92aecb" path="/var/lib/kubelet/pods/b7b22c85-eb65-412d-8cc8-48dd9e92aecb/volumes" Nov 03 20:02:52.234606 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:52.229885 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:02:52.604224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.604185 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t"] Nov 03 20:02:52.605675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.605648 4314 cpu_manager.go:414] "containerMap: removing container" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" Nov 03 20:02:52.605937 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.605904 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" Nov 03 20:02:52.606038 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.606028 4314 cpu_manager.go:414] "containerMap: removing container" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" Nov 03 20:02:52.607510 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.607486 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" Nov 03 20:02:52.607910 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.607755 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" Nov 03 20:02:52.608048 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608032 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" Nov 03 20:02:52.608149 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608139 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" Nov 03 20:02:52.608220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608210 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" Nov 03 20:02:52.608305 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608296 4314 cpu_manager.go:414] "containerMap: removing container" podUID="802abf8e-180b-4020-af28-64a5010ef933" containerName="revoke-keys" Nov 03 20:02:52.608372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608363 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802abf8e-180b-4020-af28-64a5010ef933" containerName="revoke-keys" Nov 03 20:02:52.608453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608444 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" Nov 03 20:02:52.608523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608514 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" Nov 03 20:02:52.608647 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608637 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b7b22c85-eb65-412d-8cc8-48dd9e92aecb" containerName="maas-api" Nov 03 20:02:52.608734 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.608724 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b22c85-eb65-412d-8cc8-48dd9e92aecb" containerName="maas-api" Nov 03 20:02:52.609135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.609116 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa79738-f4df-4b25-9319-ab36d813b7b9" containerName="main" Nov 03 20:02:52.609274 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.609263 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="202ad091-28ed-41c2-aeae-d60761a4d034" containerName="istio-proxy" Nov 03 20:02:52.609354 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.609346 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a318f53f-7b2c-4b68-96e9-51fcd403c2d6" containerName="main" Nov 03 20:02:52.609435 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.609426 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0209b03-fcc6-4a8c-97b8-edefc36eb120" containerName="istio-proxy" Nov 03 20:02:52.609513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.609504 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="802abf8e-180b-4020-af28-64a5010ef933" containerName="revoke-keys" Nov 03 20:02:52.609608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.609576 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e969e0-d5d8-42d3-a243-fdba3ec8534f" containerName="istio-proxy" Nov 03 20:02:52.609701 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.609691 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b22c85-eb65-412d-8cc8-48dd9e92aecb" containerName="maas-api" Nov 03 20:02:52.628604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.628547 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.633900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.633872 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dce558-openshift-default-dockercfg-rshjr\"" Nov 03 20:02:52.887147 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.887053 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.887391 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.887370 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.887520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.887504 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.887801 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.887785 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f0f2cb-9364-4969-aaca-da82fe5dd836-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.887920 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.887906 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.888014 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.887999 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx2vc\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-kube-api-access-lx2vc\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.888115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.888101 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.888233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.888220 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.888475 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.888457 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.914372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.914343 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t"] Nov 03 20:02:52.990756 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.990709 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f0f2cb-9364-4969-aaca-da82fe5dd836-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991485 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991462 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991487 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx2vc\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-kube-api-access-lx2vc\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991506 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991543 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991420 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f0f2cb-9364-4969-aaca-da82fe5dd836-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991866 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991681 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991866 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991808 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991866 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991831 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.991866 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991858 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.992082 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.991912 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.992172 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.992154 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.992364 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.992339 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.992629 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.992605 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.994564 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.994539 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.996200 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.996179 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.999719 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.999691 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:52.999989 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:52.999967 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx2vc\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-kube-api-access-lx2vc\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:53.223326 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:53.223242 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:53.590240 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:53.590210 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t"] Nov 03 20:02:53.591719 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:02:53.591690 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f0f2cb_9364_4969_aaca_da82fe5dd836.slice/crio-9e2a208f7cb9ddbcbbe76d356e5ee18411670005609197f70567ff39023ebf50 WatchSource:0}: Error finding container 9e2a208f7cb9ddbcbbe76d356e5ee18411670005609197f70567ff39023ebf50: Status 404 returned error can't find the container with id 9e2a208f7cb9ddbcbbe76d356e5ee18411670005609197f70567ff39023ebf50 Nov 03 20:02:53.594206 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:53.594172 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:53.594282 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:53.594240 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:53.594318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:53.594272 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:54.060359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.059564 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" event={"ID":"85f0f2cb-9364-4969-aaca-da82fe5dd836","Type":"ContainerStarted","Data":"f11f1f8f5bfbbdacd77940fdd119b174e4de9db4215641c0862af2a5d7ec4ac2"} Nov 03 20:02:54.060359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.059668 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" event={"ID":"85f0f2cb-9364-4969-aaca-da82fe5dd836","Type":"ContainerStarted","Data":"9e2a208f7cb9ddbcbbe76d356e5ee18411670005609197f70567ff39023ebf50"} Nov 03 20:02:54.092791 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.092724 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podStartSLOduration=2.092705721 podStartE2EDuration="2.092705721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:54.08403305 +0000 UTC m=+1881.788872391" watchObservedRunningTime="2026-11-03 20:02:54.092705721 +0000 UTC m=+1881.797545062" Nov 03 20:02:54.223591 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.223541 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:54.232291 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.232263 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:54.412684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.411012 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s"] Nov 03 20:02:54.412684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.411269 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" containerID="cri-o://d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac" gracePeriod=30 Nov 03 20:02:54.418356 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.418320 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm"] Nov 03 20:02:54.418557 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:54.418535 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" containerID="cri-o://487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7" gracePeriod=30 Nov 03 20:02:55.070290 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.070246 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:55.071477 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.071452 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:02:55.396065 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.396032 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 20:02:55.439427 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.439384 4314 patch_prober.go:29] interesting pod/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:02:55.439641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.439487 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:02:55.521273 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521235 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-envoy\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521461 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521334 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istiod-ca-cert\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521461 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521362 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-socket\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521571 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521532 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-certs\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521695 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521638 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-kube-api-access-t5r9q\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-kube-api-access-t5r9q\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521695 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521679 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-podinfo\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521812 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-credential-socket\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521985 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521904 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-token\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-token\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.521985 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.521937 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-data\") pod \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\" (UID: \"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a\") " Nov 03 20:02:55.525182 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.524279 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istiod-ca-cert" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:55.525182 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.524534 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-socket" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:55.525182 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.524709 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-certs" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:55.525415 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.525324 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-credential-socket" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:55.526074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.525940 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-data" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:55.527291 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.527254 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-podinfo" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:02:55.527971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.527940 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-envoy" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:55.531944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.531922 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-token" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:55.532049 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.532032 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-kube-api-access-t5r9q" pod "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" (UID: "b9892a5d-a52f-4f6a-b5ae-f8d926cb283a"). InnerVolumeSpecName "kube-api-access-t5r9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:55.639660 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.625081 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649664 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649609 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649693 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649711 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649726 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649739 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649753 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5r9q\" (UniqueName: \"kubernetes.io/projected/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-kube-api-access-t5r9q\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649772 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:55.649851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:55.649790 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:56.080012 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.079980 4314 generic.go:356] "Generic (PLEG): container finished" podID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerID="3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301" exitCode=0 Nov 03 20:02:56.080468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.080049 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" Nov 03 20:02:56.080468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.080075 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" event={"ID":"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a","Type":"ContainerDied","Data":"3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301"} Nov 03 20:02:56.080468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.080134 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts" event={"ID":"b9892a5d-a52f-4f6a-b5ae-f8d926cb283a","Type":"ContainerDied","Data":"e7c608eaaf8abdc8cff9ee71b2d720bb74db6e7602f8b7d39eb8c215cd19ed1c"} Nov 03 20:02:56.080468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.080157 4314 scope.go:122] "RemoveContainer" containerID="3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301" Nov 03 20:02:56.093696 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.093677 4314 scope.go:122] "RemoveContainer" containerID="3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301" Nov 03 20:02:56.093986 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:02:56.093962 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301\": container with ID starting with 3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301 not found: ID does not exist" containerID="3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301" Nov 03 20:02:56.094074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.093993 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301"} err="failed to get container status \"3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301\": rpc error: code = NotFound desc = could not find container \"3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301\": container with ID starting with 3b20a585dab9ada52a0773b2c825d46b9969c5a1af7f7e141e43c20eeb7e0301 not found: ID does not exist" Nov 03 20:02:56.110033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.110003 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts"] Nov 03 20:02:56.114894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.114868 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-rcbts"] Nov 03 20:02:56.430516 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:56.430431 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" path="/var/lib/kubelet/pods/b9892a5d-a52f-4f6a-b5ae-f8d926cb283a/volumes" Nov 03 20:02:57.157636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:57.156780 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8"] Nov 03 20:02:57.158123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:57.157408 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" containerID="cri-o://5fe2a850e7bfa3446eb624f0f7f06efda8dc8ddc0c72f0f38a9e08111dbdfb17" gracePeriod=30 Nov 03 20:02:58.382602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.381914 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth"] Nov 03 20:02:58.388601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.383198 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" Nov 03 20:02:58.388601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.383223 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" Nov 03 20:02:58.388601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.383458 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9892a5d-a52f-4f6a-b5ae-f8d926cb283a" containerName="istio-proxy" Nov 03 20:02:58.395628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.390701 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.405603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.400875 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth"] Nov 03 20:02:58.458835 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.458801 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-801162-openshift-default-dockercfg-2wrnp\"" Nov 03 20:02:58.544858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.544821 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-certs\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.545200 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.545175 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-podinfo\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.545434 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.545416 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc2h\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-kube-api-access-7rc2h\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.568604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.545663 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-envoy\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.571215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.571179 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istiod-ca-cert\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.571496 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.571462 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-data\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.572215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.572190 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-token\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.572596 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.572395 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-socket\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.573217 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.573198 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-credential-socket\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680290 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-podinfo\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680417 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc2h\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-kube-api-access-7rc2h\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680527 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-envoy\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680633 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istiod-ca-cert\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680666 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-data\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680691 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-token\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680735 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-socket\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680847 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-credential-socket\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.680924 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-certs\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.704662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.681459 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-certs\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.723608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.719674 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-data\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.723608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.720309 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-socket\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.723608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.720667 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-credential-socket\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.756557 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.739571 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istiod-ca-cert\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.756557 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.740355 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-envoy\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.756557 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.756461 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-token\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.804974 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.793447 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-podinfo\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:58.857058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:58.857022 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc2h\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-kube-api-access-7rc2h\") pod \"e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:59.013482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:59.013451 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:02:59.364546 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:59.364513 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth"] Nov 03 20:02:59.364724 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:02:59.364628 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6612b77c_c2f2_4eb4_a203_2ab47761d23b.slice/crio-8d5eff92c0330e08fc3401710c6829fcfcbde14ce6906713d8b405a478d8e906 WatchSource:0}: Error finding container 8d5eff92c0330e08fc3401710c6829fcfcbde14ce6906713d8b405a478d8e906: Status 404 returned error can't find the container with id 8d5eff92c0330e08fc3401710c6829fcfcbde14ce6906713d8b405a478d8e906 Nov 03 20:02:59.368386 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:59.368345 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:59.368472 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:59.368434 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:59.368519 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:59.368477 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:02:59.954605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:02:59.952057 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 20:03:00.013169 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013129 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-data\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013214 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-token\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-token\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013254 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-envoy\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013343 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-certs\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013488 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013390 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9521c02-9805-49dc-ace6-032e561bd5c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9521c02-9805-49dc-ace6-032e561bd5c6-istiod-ca-cert\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013488 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013467 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-podinfo\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013627 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013610 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-kube-api-access-xj4vl\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-kube-api-access-xj4vl\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013754 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013733 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-socket\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.013842 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.013826 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-credential-socket\") pod \"d9521c02-9805-49dc-ace6-032e561bd5c6\" (UID: \"d9521c02-9805-49dc-ace6-032e561bd5c6\") " Nov 03 20:03:00.014071 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.014047 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-certs" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.014652 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.014624 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-credential-socket" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.015037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.015015 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.015128 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.015045 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.015278 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.015235 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-socket" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.015690 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.015637 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9521c02-9805-49dc-ace6-032e561bd5c6-istiod-ca-cert" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:00.015859 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.015837 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-data" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.017011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.016978 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-kube-api-access-xj4vl" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "kube-api-access-xj4vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:00.017173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.017148 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-podinfo" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:03:00.018257 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.018232 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-envoy" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.018815 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.018780 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-token" pod "d9521c02-9805-49dc-ace6-032e561bd5c6" (UID: "d9521c02-9805-49dc-ace6-032e561bd5c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:00.087185 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.086892 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 20:03:00.116479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116084 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-token\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-token\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.116479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116232 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-credential-socket\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.116479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116270 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503264b1-ec82-44d1-ab06-e466c4c9827e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503264b1-ec82-44d1-ab06-e466c4c9827e-istiod-ca-cert\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.116479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116345 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-certs\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116538 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-envoy\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116633 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-data\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116665 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-kube-api-access-4hzps\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-kube-api-access-4hzps\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116716 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-podinfo\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.116747 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-socket\") pod \"503264b1-ec82-44d1-ab06-e466c4c9827e\" (UID: \"503264b1-ec82-44d1-ab06-e466c4c9827e\") " Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117436 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117460 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj4vl\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-kube-api-access-xj4vl\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117477 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117493 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117506 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117519 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9521c02-9805-49dc-ace6-032e561bd5c6-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117531 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9521c02-9805-49dc-ace6-032e561bd5c6-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.118369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.117975 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-socket" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.119062 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.118881 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-credential-socket" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.119120 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.119080 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-certs" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.119542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.119519 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503264b1-ec82-44d1-ab06-e466c4c9827e-istiod-ca-cert" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:00.119782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.119762 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-data" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.122553 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.122531 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-kube-api-access-4hzps" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "kube-api-access-4hzps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:00.122672 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.122570 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-token" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:00.124037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.124012 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-envoy" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:00.124445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.124423 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-podinfo" pod "503264b1-ec82-44d1-ab06-e466c4c9827e" (UID: "503264b1-ec82-44d1-ab06-e466c4c9827e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:03:00.219369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219286 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219332 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hzps\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-kube-api-access-4hzps\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219349 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219363 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219378 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219392 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219406 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503264b1-ec82-44d1-ab06-e466c4c9827e-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219419 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.219626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.219432 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503264b1-ec82-44d1-ab06-e466c4c9827e-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:00.243559 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.243526 4314 generic.go:356] "Generic (PLEG): container finished" podID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerID="d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac" exitCode=0 Nov 03 20:03:00.243773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.243576 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" Nov 03 20:03:00.243773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.243617 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" event={"ID":"d9521c02-9805-49dc-ace6-032e561bd5c6","Type":"ContainerDied","Data":"d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac"} Nov 03 20:03:00.243773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.243673 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s" event={"ID":"d9521c02-9805-49dc-ace6-032e561bd5c6","Type":"ContainerDied","Data":"2ab8075f52a4ab0a9d4a82a0b7da041beec497fbc354d938cce85bda1c4e14b3"} Nov 03 20:03:00.243773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.243695 4314 scope.go:122] "RemoveContainer" containerID="d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac" Nov 03 20:03:00.247687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.247660 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" event={"ID":"6612b77c-c2f2-4eb4-a203-2ab47761d23b","Type":"ContainerStarted","Data":"24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937"} Nov 03 20:03:00.247811 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.247708 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" event={"ID":"6612b77c-c2f2-4eb4-a203-2ab47761d23b","Type":"ContainerStarted","Data":"8d5eff92c0330e08fc3401710c6829fcfcbde14ce6906713d8b405a478d8e906"} Nov 03 20:03:00.250011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.249987 4314 generic.go:356] "Generic (PLEG): container finished" podID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerID="487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7" exitCode=0 Nov 03 20:03:00.250152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.250012 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" event={"ID":"503264b1-ec82-44d1-ab06-e466c4c9827e","Type":"ContainerDied","Data":"487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7"} Nov 03 20:03:00.250152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.250037 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" Nov 03 20:03:00.250152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.250064 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm" event={"ID":"503264b1-ec82-44d1-ab06-e466c4c9827e","Type":"ContainerDied","Data":"d98599fbafc83627a2ebe9c623a06b2aa3264a06a6841b8442ff5aadef4a80a3"} Nov 03 20:03:00.260988 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.260292 4314 scope.go:122] "RemoveContainer" containerID="d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac" Nov 03 20:03:00.260988 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:00.260599 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac\": container with ID starting with d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac not found: ID does not exist" containerID="d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac" Nov 03 20:03:00.260988 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.260632 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac"} err="failed to get container status \"d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac\": rpc error: code = NotFound desc = could not find container \"d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac\": container with ID starting with d0f49115d8441db534ba5b6c7c50ac53fc5920a14d10cda51521cd44338290ac not found: ID does not exist" Nov 03 20:03:00.260988 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.260657 4314 scope.go:122] "RemoveContainer" containerID="487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7" Nov 03 20:03:00.271934 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.271887 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podStartSLOduration=2.271875577 podStartE2EDuration="2.271875577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:00.266195114 +0000 UTC m=+1887.971034466" watchObservedRunningTime="2026-11-03 20:03:00.271875577 +0000 UTC m=+1887.976714993" Nov 03 20:03:00.278024 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.278006 4314 scope.go:122] "RemoveContainer" containerID="763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca" Nov 03 20:03:00.293785 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.293754 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm"] Nov 03 20:03:00.293886 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.293803 4314 scope.go:122] "RemoveContainer" containerID="487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7" Nov 03 20:03:00.294168 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:00.294144 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7\": container with ID starting with 487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7 not found: ID does not exist" containerID="487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7" Nov 03 20:03:00.294226 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.294179 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7"} err="failed to get container status \"487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7\": rpc error: code = NotFound desc = could not find container \"487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7\": container with ID starting with 487465d57fcbca2984bba3131e61111cb344f993f86c22b720be7de8c639f4b7 not found: ID does not exist" Nov 03 20:03:00.294226 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.294202 4314 scope.go:122] "RemoveContainer" containerID="763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca" Nov 03 20:03:00.294661 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:00.294634 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca\": container with ID starting with 763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca not found: ID does not exist" containerID="763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca" Nov 03 20:03:00.294856 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.294831 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca"} err="failed to get container status \"763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca\": rpc error: code = NotFound desc = could not find container \"763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca\": container with ID starting with 763a9e341fc1c6ceea0a7429b68c9733e3429575d3515d9f555ed9b6f5a5daca not found: ID does not exist" Nov 03 20:03:00.303347 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.303264 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-wnbmm"] Nov 03 20:03:00.313225 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.313199 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s"] Nov 03 20:03:00.318104 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.318052 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-225668-openshift-default-765b499d89-qjf5s"] Nov 03 20:03:00.445705 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.445670 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" path="/var/lib/kubelet/pods/503264b1-ec82-44d1-ab06-e466c4c9827e/volumes" Nov 03 20:03:00.446336 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:00.446320 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" path="/var/lib/kubelet/pods/d9521c02-9805-49dc-ace6-032e561bd5c6/volumes" Nov 03 20:03:01.014191 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:01.014155 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:03:01.020780 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:01.020754 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:03:01.260212 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:01.260180 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:03:01.261051 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:01.261030 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:03:02.277453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.275519 4314 generic.go:356] "Generic (PLEG): container finished" podID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerID="5fe2a850e7bfa3446eb624f0f7f06efda8dc8ddc0c72f0f38a9e08111dbdfb17" exitCode=0 Nov 03 20:03:02.277453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.276783 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" event={"ID":"db0a4afc-942b-4d68-8eec-86290a693e1a","Type":"ContainerDied","Data":"5fe2a850e7bfa3446eb624f0f7f06efda8dc8ddc0c72f0f38a9e08111dbdfb17"} Nov 03 20:03:02.277453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.276842 4314 scope.go:122] "RemoveContainer" containerID="47a961f8e87b41f26476d018eefa8846721d4001cee96b54b4c2af802e36f601" Nov 03 20:03:02.406868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.406839 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 20:03:02.466658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.466352 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-certs\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.466658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.466433 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-token\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.466658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.466478 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-credential-socket\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.466658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.466567 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-podinfo\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.467044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.466771 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-socket\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.467044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.466825 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-envoy\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.467188 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.467048 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0a4afc-942b-4d68-8eec-86290a693e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0a4afc-942b-4d68-8eec-86290a693e1a-istiod-ca-cert\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.467188 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.467095 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-kube-api-access-p9zzb\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-kube-api-access-p9zzb\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.467188 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.467139 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-data\") pod \"db0a4afc-942b-4d68-8eec-86290a693e1a\" (UID: \"db0a4afc-942b-4d68-8eec-86290a693e1a\") " Nov 03 20:03:02.470033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.469445 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-data" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:02.473916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.472661 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-socket" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:02.473916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.472688 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0a4afc-942b-4d68-8eec-86290a693e1a-istiod-ca-cert" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:02.473916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.472863 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-certs" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:02.475102 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.475049 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-credential-socket" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:02.521097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.517445 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-kube-api-access-p9zzb" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "kube-api-access-p9zzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:02.521097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.519449 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-token" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:02.521097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.521001 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-podinfo" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:03:02.521512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.521447 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-envoy" pod "db0a4afc-942b-4d68-8eec-86290a693e1a" (UID: "db0a4afc-942b-4d68-8eec-86290a693e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:02.575913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.575875 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.575927 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.575944 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0a4afc-942b-4d68-8eec-86290a693e1a-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.575959 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zzb\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-kube-api-access-p9zzb\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.575975 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.575988 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.576002 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.576016 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0a4afc-942b-4d68-8eec-86290a693e1a-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:02.576097 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:02.576030 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0a4afc-942b-4d68-8eec-86290a693e1a-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:03.287542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:03.287499 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" event={"ID":"db0a4afc-942b-4d68-8eec-86290a693e1a","Type":"ContainerDied","Data":"0eef96424a8f34ebe086f0bc3dee942c469dcff8d987613af4691822a0ad7de9"} Nov 03 20:03:03.288163 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:03.287650 4314 scope.go:122] "RemoveContainer" containerID="5fe2a850e7bfa3446eb624f0f7f06efda8dc8ddc0c72f0f38a9e08111dbdfb17" Nov 03 20:03:03.288163 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:03.287847 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8" Nov 03 20:03:03.318356 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:03.318333 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8"] Nov 03 20:03:03.327072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:03.327048 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-l67n8"] Nov 03 20:03:03.679136 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:03.679100 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb"] Nov 03 20:03:03.679363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:03.679310 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" podUID="a33d9e6b-a0d3-491e-b41b-682cfe814fb4" containerName="maas-api" containerID="cri-o://24607051607b14a9afc529ad512d12762fb2d391308f7d5431632adfdc9c1787" gracePeriod=30 Nov 03 20:03:04.199093 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.192166 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg"] Nov 03 20:03:04.233428 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.233358 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-102379-9dqhg"] Nov 03 20:03:04.360604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.357065 4314 generic.go:356] "Generic (PLEG): container finished" podID="a33d9e6b-a0d3-491e-b41b-682cfe814fb4" containerID="24607051607b14a9afc529ad512d12762fb2d391308f7d5431632adfdc9c1787" exitCode=0 Nov 03 20:03:04.360604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.357345 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" event={"ID":"a33d9e6b-a0d3-491e-b41b-682cfe814fb4","Type":"ContainerDied","Data":"24607051607b14a9afc529ad512d12762fb2d391308f7d5431632adfdc9c1787"} Nov 03 20:03:04.438454 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.438417 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e67e41-fb69-412b-aa31-03c50976ef72" path="/var/lib/kubelet/pods/d5e67e41-fb69-412b-aa31-03c50976ef72/volumes" Nov 03 20:03:04.440619 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.440575 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" path="/var/lib/kubelet/pods/db0a4afc-942b-4d68-8eec-86290a693e1a/volumes" Nov 03 20:03:04.468343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.468288 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 20:03:04.547531 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.547498 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-openshift-service-ca\") pod \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " Nov 03 20:03:04.547759 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.547539 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-maas-api-tls\") pod \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " Nov 03 20:03:04.547759 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.547669 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-kube-api-access-4hbpj\" (UniqueName: \"kubernetes.io/projected/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-kube-api-access-4hbpj\") pod \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " Nov 03 20:03:04.547899 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.547883 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-metrics-certs\") pod \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\" (UID: \"a33d9e6b-a0d3-491e-b41b-682cfe814fb4\") " Nov 03 20:03:04.548078 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.548048 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-openshift-service-ca" pod "a33d9e6b-a0d3-491e-b41b-682cfe814fb4" (UID: "a33d9e6b-a0d3-491e-b41b-682cfe814fb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:04.548726 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.548698 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:04.549994 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.549969 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-kube-api-access-4hbpj" pod "a33d9e6b-a0d3-491e-b41b-682cfe814fb4" (UID: "a33d9e6b-a0d3-491e-b41b-682cfe814fb4"). InnerVolumeSpecName "kube-api-access-4hbpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:04.550315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.550295 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-metrics-certs" pod "a33d9e6b-a0d3-491e-b41b-682cfe814fb4" (UID: "a33d9e6b-a0d3-491e-b41b-682cfe814fb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:04.550388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.550317 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-maas-api-tls" pod "a33d9e6b-a0d3-491e-b41b-682cfe814fb4" (UID: "a33d9e6b-a0d3-491e-b41b-682cfe814fb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:04.650293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.650258 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:04.650293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.650292 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hbpj\" (UniqueName: \"kubernetes.io/projected/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-kube-api-access-4hbpj\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:04.650498 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:04.650310 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a33d9e6b-a0d3-491e-b41b-682cfe814fb4-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:05.383894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:05.382759 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" event={"ID":"a33d9e6b-a0d3-491e-b41b-682cfe814fb4","Type":"ContainerDied","Data":"2e8ed197649c8f597d1c7174ab3a9470881681cd79e34a7129ea1439613d0655"} Nov 03 20:03:05.383894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:05.382838 4314 scope.go:122] "RemoveContainer" containerID="24607051607b14a9afc529ad512d12762fb2d391308f7d5431632adfdc9c1787" Nov 03 20:03:05.383894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:05.382961 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" Nov 03 20:03:05.428073 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:05.428037 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb"] Nov 03 20:03:05.432136 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:05.432112 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb"] Nov 03 20:03:06.434704 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:06.434672 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33d9e6b-a0d3-491e-b41b-682cfe814fb4" path="/var/lib/kubelet/pods/a33d9e6b-a0d3-491e-b41b-682cfe814fb4/volumes" Nov 03 20:03:07.310253 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:07.310208 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-102379-65549b87c-wzvpb" podUID="a33d9e6b-a0d3-491e-b41b-682cfe814fb4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.53:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:10.988654 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:10.987271 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7"] Nov 03 20:03:10.988654 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:10.987536 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" containerID="cri-o://ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107" gracePeriod=30 Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.715469 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx"] Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716635 4314 cpu_manager.go:414] "containerMap: removing container" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716656 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716671 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a33d9e6b-a0d3-491e-b41b-682cfe814fb4" containerName="maas-api" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716679 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33d9e6b-a0d3-491e-b41b-682cfe814fb4" containerName="maas-api" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716727 4314 cpu_manager.go:414] "containerMap: removing container" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716736 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716759 4314 cpu_manager.go:414] "containerMap: removing container" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716767 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716979 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.716993 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9521c02-9805-49dc-ace6-032e561bd5c6" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.717019 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.717029 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33d9e6b-a0d3-491e-b41b-682cfe814fb4" containerName="maas-api" Nov 03 20:03:14.718600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.717042 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" Nov 03 20:03:14.728075 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.726871 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.732211 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.732176 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d12868-openshift-default-dockercfg-vq2jt\"" Nov 03 20:03:14.744655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.743326 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx"] Nov 03 20:03:14.783311 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.783262 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-credential-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.783677 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.783649 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwwd\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-kube-api-access-mxwwd\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.783849 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.783828 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-data\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.784044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.784024 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40337036-b540-4958-af83-99d1c44752c4-istiod-ca-cert\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.784229 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.784211 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-envoy\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.784347 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.784332 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.784488 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.784473 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40337036-b540-4958-af83-99d1c44752c4-istio-podinfo\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.784735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.784713 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-certs\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.784898 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.784879 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-istio-token\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.886895 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.886860 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-envoy\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.886895 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.886899 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.886948 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40337036-b540-4958-af83-99d1c44752c4-istio-podinfo\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887014 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-certs\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887038 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-istio-token\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887176 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-credential-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887214 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwwd\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-kube-api-access-mxwwd\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887253 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-data\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887349 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40337036-b540-4958-af83-99d1c44752c4-istiod-ca-cert\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887653 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887544 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-certs\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887653 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887344 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887861 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887841 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-data\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.887952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.887930 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-credential-socket\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.888564 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.888542 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40337036-b540-4958-af83-99d1c44752c4-istiod-ca-cert\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.889741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.889708 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-envoy\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.890108 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.890087 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40337036-b540-4958-af83-99d1c44752c4-istio-podinfo\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.895804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.895771 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-istio-token\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:14.896132 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:14.896106 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwwd\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-kube-api-access-mxwwd\") pod \"e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:15.063971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:15.063935 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:15.415801 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:15.415748 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:15.416048 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:15.416007 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:15.416252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:15.416223 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:15.418044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:15.418016 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx"] Nov 03 20:03:15.518155 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:15.518126 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" event={"ID":"40337036-b540-4958-af83-99d1c44752c4","Type":"ContainerStarted","Data":"f607bd87a97cccb433ffc9cb401aee143eb03cdd436378eac02f49353500fd21"} Nov 03 20:03:16.506019 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.505971 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.526656 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn"] Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.527887 4314 cpu_manager.go:414] "containerMap: removing container" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.527908 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.527933 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.527941 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.528004 4314 cpu_manager.go:414] "containerMap: removing container" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.528012 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0a4afc-942b-4d68-8eec-86290a693e1a" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.528197 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.528218 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" Nov 03 20:03:16.529279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.528227 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="503264b1-ec82-44d1-ab06-e466c4c9827e" containerName="istio-proxy" Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.532939 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-podinfo\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533075 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-data\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533156 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-envoy\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533244 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-socket\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533312 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-token\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-token\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533368 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istiod-ca-cert\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533413 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-credential-socket\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533472 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-kube-api-access-lm7mb\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-kube-api-access-lm7mb\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.533609 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-certs\") pod \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\" (UID: \"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba\") " Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.534510 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-certs" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.534835 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istiod-ca-cert" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.535010 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-data" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.535068 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-credential-socket" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:16.536286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.535482 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-socket" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:16.537176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.536884 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-token" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:16.553123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.552906 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-envoy" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:16.553276 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.553237 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-podinfo" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:03:16.556305 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.556127 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn"] Nov 03 20:03:16.556305 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.556255 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.563675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.560697 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:03:16.563675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.560758 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:16.604648 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.602441 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-kube-api-access-lm7mb" pod "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" (UID: "7d9cbe6d-f94f-4d28-95df-a55517f3c4ba"). InnerVolumeSpecName "kube-api-access-lm7mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:16.625426 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.625042 4314 generic.go:356] "Generic (PLEG): container finished" podID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerID="ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107" exitCode=0 Nov 03 20:03:16.625426 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.625377 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" Nov 03 20:03:16.626482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.626450 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" event={"ID":"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba","Type":"ContainerDied","Data":"ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107"} Nov 03 20:03:16.626712 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.626692 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7" event={"ID":"7d9cbe6d-f94f-4d28-95df-a55517f3c4ba","Type":"ContainerDied","Data":"1ba2b5c1eb3b1fb9a9fb281d81158f0d88f99a887ab07f05bfcfdcc3a747c1e2"} Nov 03 20:03:16.626844 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.626831 4314 scope.go:122] "RemoveContainer" containerID="ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107" Nov 03 20:03:16.633914 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.633885 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" event={"ID":"40337036-b540-4958-af83-99d1c44752c4","Type":"ContainerStarted","Data":"73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c"} Nov 03 20:03:16.635426 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.635398 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.635632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.635612 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.635794 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.635778 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrgf\" (UniqueName: \"kubernetes.io/projected/a1b3b538-8dc4-418d-876b-f64f03e4b21f-kube-api-access-gnrgf\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.636023 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636006 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636147 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636271 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636257 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7mb\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-kube-api-access-lm7mb\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636357 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636433 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636499 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636491 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636556 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636549 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636637 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636624 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.636727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.636718 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:16.683627 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.681791 4314 scope.go:122] "RemoveContainer" containerID="7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e" Nov 03 20:03:16.772605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.764092 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podStartSLOduration=2.764070528 podStartE2EDuration="2.764070528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:16.679852212 +0000 UTC m=+1904.384691552" watchObservedRunningTime="2026-11-03 20:03:16.764070528 +0000 UTC m=+1904.468909870" Nov 03 20:03:16.772605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.770032 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.772605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.770096 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.817058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.792269 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.817058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.793769 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrgf\" (UniqueName: \"kubernetes.io/projected/a1b3b538-8dc4-418d-876b-f64f03e4b21f-kube-api-access-gnrgf\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.817058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.804907 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7"] Nov 03 20:03:16.817058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.806980 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-dzxw7"] Nov 03 20:03:16.837612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.835455 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrgf\" (UniqueName: \"kubernetes.io/projected/a1b3b538-8dc4-418d-876b-f64f03e4b21f-kube-api-access-gnrgf\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.851298 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.850102 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936a5ff2-nqpqn\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:16.882812 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.878779 4314 scope.go:122] "RemoveContainer" containerID="ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107" Nov 03 20:03:16.882812 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:16.879984 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107\": container with ID starting with ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107 not found: ID does not exist" containerID="ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107" Nov 03 20:03:16.882812 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.880028 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107"} err="failed to get container status \"ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107\": rpc error: code = NotFound desc = could not find container \"ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107\": container with ID starting with ff949455fb47be3016f35ef52658abdddffe088f723cdd621de0c8839bf26107 not found: ID does not exist" Nov 03 20:03:16.882812 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.880054 4314 scope.go:122] "RemoveContainer" containerID="7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e" Nov 03 20:03:16.882812 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:16.880386 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e\": container with ID starting with 7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e not found: ID does not exist" containerID="7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e" Nov 03 20:03:16.882812 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.880412 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e"} err="failed to get container status \"7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e\": rpc error: code = NotFound desc = could not find container \"7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e\": container with ID starting with 7ec5dc3e967bc1a318dfabb5b90d66a56a1ee5049e22644cf4c7b0783a38f74e not found: ID does not exist" Nov 03 20:03:16.929646 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:16.926577 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:17.090790 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:17.090659 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:17.100175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:17.097778 4314 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:03:17.100175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:17.097840 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:03:17.551825 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:17.545977 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn"] Nov 03 20:03:17.652911 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:17.652870 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" event={"ID":"a1b3b538-8dc4-418d-876b-f64f03e4b21f","Type":"ContainerStarted","Data":"f288884a71558e011a076bfd71300c71bd8677efda68cd479dded66808a8efce"} Nov 03 20:03:18.081604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:18.080836 4314 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:03:18.081604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:18.080899 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:03:18.432613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:18.432546 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" path="/var/lib/kubelet/pods/7d9cbe6d-f94f-4d28-95df-a55517f3c4ba/volumes" Nov 03 20:03:18.689254 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:18.689215 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" event={"ID":"a1b3b538-8dc4-418d-876b-f64f03e4b21f","Type":"ContainerStarted","Data":"8bd2371e945eadd704f349ba09c86d65def4a3e8436150f4af8e69e7b4c1d476"} Nov 03 20:03:18.706326 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:18.705674 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" podStartSLOduration=2.705655279 podStartE2EDuration="2.705655279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:18.704560281 +0000 UTC m=+1906.409399623" watchObservedRunningTime="2026-11-03 20:03:18.705655279 +0000 UTC m=+1906.410494619" Nov 03 20:03:19.100602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:19.074995 4314 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:03:19.100602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:19.075084 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:03:19.706187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:19.706149 4314 generic.go:356] "Generic (PLEG): container finished" podID="a1b3b538-8dc4-418d-876b-f64f03e4b21f" containerID="8bd2371e945eadd704f349ba09c86d65def4a3e8436150f4af8e69e7b4c1d476" exitCode=0 Nov 03 20:03:19.706783 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:19.706223 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" event={"ID":"a1b3b538-8dc4-418d-876b-f64f03e4b21f","Type":"ContainerDied","Data":"8bd2371e945eadd704f349ba09c86d65def4a3e8436150f4af8e69e7b4c1d476"} Nov 03 20:03:20.065605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:20.064931 4314 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:03:20.065605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:20.064996 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:03:20.952254 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:20.952224 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:21.013567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.013532 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-openshift-ca\") pod \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " Nov 03 20:03:21.013833 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.013809 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-xks-ca\") pod \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " Nov 03 20:03:21.013966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.013939 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-openshift-ca" pod "a1b3b538-8dc4-418d-876b-f64f03e4b21f" (UID: "a1b3b538-8dc4-418d-876b-f64f03e4b21f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:21.014038 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.014006 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b3b538-8dc4-418d-876b-f64f03e4b21f-kube-api-access-gnrgf\" (UniqueName: \"kubernetes.io/projected/a1b3b538-8dc4-418d-876b-f64f03e4b21f-kube-api-access-gnrgf\") pod \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\" (UID: \"a1b3b538-8dc4-418d-876b-f64f03e4b21f\") " Nov 03 20:03:21.014799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.014773 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.017644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.016511 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b3b538-8dc4-418d-876b-f64f03e4b21f-kube-api-access-gnrgf" pod "a1b3b538-8dc4-418d-876b-f64f03e4b21f" (UID: "a1b3b538-8dc4-418d-876b-f64f03e4b21f"). InnerVolumeSpecName "kube-api-access-gnrgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:21.018061 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.018031 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-xks-ca" pod "a1b3b538-8dc4-418d-876b-f64f03e4b21f" (UID: "a1b3b538-8dc4-418d-876b-f64f03e4b21f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:21.092145 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.091701 4314 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:03:21.092145 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.091777 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:03:21.116514 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.116473 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b3b538-8dc4-418d-876b-f64f03e4b21f-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.116514 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.116506 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnrgf\" (UniqueName: \"kubernetes.io/projected/a1b3b538-8dc4-418d-876b-f64f03e4b21f-kube-api-access-gnrgf\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:21.726681 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.726647 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk"] Nov 03 20:03:21.744333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.737622 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a1b3b538-8dc4-418d-876b-f64f03e4b21f" containerName="revoke-keys" Nov 03 20:03:21.744333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.737653 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b3b538-8dc4-418d-876b-f64f03e4b21f" containerName="revoke-keys" Nov 03 20:03:21.744333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.737726 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" Nov 03 20:03:21.744333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.737735 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9cbe6d-f94f-4d28-95df-a55517f3c4ba" containerName="istio-proxy" Nov 03 20:03:21.744333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.737919 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b3b538-8dc4-418d-876b-f64f03e4b21f" containerName="revoke-keys" Nov 03 20:03:21.744333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.738749 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" Nov 03 20:03:21.750047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.749024 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn" event={"ID":"a1b3b538-8dc4-418d-876b-f64f03e4b21f","Type":"ContainerDied","Data":"f288884a71558e011a076bfd71300c71bd8677efda68cd479dded66808a8efce"} Nov 03 20:03:21.750047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.749071 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f288884a71558e011a076bfd71300c71bd8677efda68cd479dded66808a8efce" Nov 03 20:03:21.750047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.749089 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk"] Nov 03 20:03:21.750047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.749181 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.754512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.754488 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-79c47f-openshift-default-dockercfg-ftrfj\"" Nov 03 20:03:21.829290 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829254 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829465 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829292 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-podinfo\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829465 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829406 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-certs\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829465 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829449 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-data\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829721 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829539 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istiod-ca-cert\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829721 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829703 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-token\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829850 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829729 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-envoy\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829850 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829769 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-credential-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.829850 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.829816 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcblz\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-kube-api-access-tcblz\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931240 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931201 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-data\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931410 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931265 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istiod-ca-cert\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931410 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931374 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-token\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931549 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931448 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-envoy\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931549 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931500 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-credential-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931549 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931536 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcblz\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-kube-api-access-tcblz\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931729 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931697 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931729 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931713 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-data\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931862 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931722 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-podinfo\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.931862 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.931825 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-certs\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.932152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.932125 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istiod-ca-cert\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.932270 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.932166 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-credential-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.932415 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.932389 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-socket\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.932706 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.932687 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-certs\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.934131 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.934104 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-envoy\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.934393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.934376 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-podinfo\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.940243 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.940177 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-token\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:21.940661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:21.940638 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcblz\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-kube-api-access-tcblz\") pod \"e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:22.034520 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:22.033993 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:03:22.064886 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.064848 4314 patch_prober.go:29] interesting pod/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:03:22.065040 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.064906 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:03:22.131605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.124899 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:22.517243 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.517204 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk"] Nov 03 20:03:22.537954 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:03:22.537882 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb7ad2f_e8e4_45e2_8b1a_7cd98bcedb74.slice/crio-56e1c2b4989ab519057384d90ccfd88820d68db990866be8c73fe72ad1b8472b WatchSource:0}: Error finding container 56e1c2b4989ab519057384d90ccfd88820d68db990866be8c73fe72ad1b8472b: Status 404 returned error can't find the container with id 56e1c2b4989ab519057384d90ccfd88820d68db990866be8c73fe72ad1b8472b Nov 03 20:03:22.542086 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.542052 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:22.542201 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.542136 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:22.542201 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.542182 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:03:22.799645 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:22.796104 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" event={"ID":"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74","Type":"ContainerStarted","Data":"56e1c2b4989ab519057384d90ccfd88820d68db990866be8c73fe72ad1b8472b"} Nov 03 20:03:23.113660 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:23.110957 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:23.113660 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:23.112729 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:23.167611 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:23.164437 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:03:23.806289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:23.806252 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" event={"ID":"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74","Type":"ContainerStarted","Data":"ff20e67978f90292e541bc338fa9f854e6fcefa1a50a6e02fa090dd366a3cff0"} Nov 03 20:03:23.831417 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:23.831357 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" podStartSLOduration=2.831340257 podStartE2EDuration="2.831340257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:23.825930325 +0000 UTC m=+1911.530769668" watchObservedRunningTime="2026-11-03 20:03:23.831340257 +0000 UTC m=+1911.536179596" Nov 03 20:03:24.130477 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:24.130404 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:24.131764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:24.131741 4314 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:03:24.131843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:24.131790 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 20:03:25.130946 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:25.130912 4314 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:03:25.131358 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:25.130969 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 20:03:26.131328 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:26.131281 4314 patch_prober.go:29] interesting pod/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:03:26.131781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:26.131343 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 20:03:26.336420 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:26.336383 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk"] Nov 03 20:03:26.846117 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:26.846075 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerName="istio-proxy" containerID="cri-o://ff20e67978f90292e541bc338fa9f854e6fcefa1a50a6e02fa090dd366a3cff0" gracePeriod=30 Nov 03 20:03:33.577483 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:33.577451 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt"] Nov 03 20:03:34.012268 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.012233 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk"] Nov 03 20:03:34.017563 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.017538 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.023364 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.023335 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk"] Nov 03 20:03:34.023820 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.023790 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:03:34.023947 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.023868 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:34.102506 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.102466 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jt7\" (UniqueName: \"kubernetes.io/projected/b152554a-9b5f-4fae-9174-0662f4eda8c5-kube-api-access-g7jt7\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.102735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.102549 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.102735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.102628 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.204170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.204134 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jt7\" (UniqueName: \"kubernetes.io/projected/b152554a-9b5f-4fae-9174-0662f4eda8c5-kube-api-access-g7jt7\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.204358 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.204229 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.204358 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.204273 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.205107 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.205075 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.206865 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.206834 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.211980 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.211953 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jt7\" (UniqueName: \"kubernetes.io/projected/b152554a-9b5f-4fae-9174-0662f4eda8c5-kube-api-access-g7jt7\") pod \"maas-api-revoke-sub-64cb1c6c-xvtjk\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.354900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.354817 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:34.652792 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.652759 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk"] Nov 03 20:03:34.943870 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.943829 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" event={"ID":"b152554a-9b5f-4fae-9174-0662f4eda8c5","Type":"ContainerStarted","Data":"0a6fea7cecf09aa32a97c6461aebcd402348e34ca69d7f361ada392c2ec693ef"} Nov 03 20:03:34.947721 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.946116 4314 generic.go:356] "Generic (PLEG): container finished" podID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerID="ff20e67978f90292e541bc338fa9f854e6fcefa1a50a6e02fa090dd366a3cff0" exitCode=0 Nov 03 20:03:34.947721 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:34.946192 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" event={"ID":"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74","Type":"ContainerDied","Data":"ff20e67978f90292e541bc338fa9f854e6fcefa1a50a6e02fa090dd366a3cff0"} Nov 03 20:03:34.951798 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:34.951754 4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb7ad2f_e8e4_45e2_8b1a_7cd98bcedb74.slice/crio-ff20e67978f90292e541bc338fa9f854e6fcefa1a50a6e02fa090dd366a3cff0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb7ad2f_e8e4_45e2_8b1a_7cd98bcedb74.slice/crio-conmon-ff20e67978f90292e541bc338fa9f854e6fcefa1a50a6e02fa090dd366a3cff0.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:03:35.051057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.051021 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:35.118846 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.118808 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istiod-ca-cert\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119045 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.118879 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-credential-socket\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119045 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.118934 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-kube-api-access-tcblz\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-kube-api-access-tcblz\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119084 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-podinfo\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119126 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-envoy\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119153 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-data\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119194 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-socket\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119237 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-credential-socket" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.119353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119278 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-certs\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119529 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119351 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-token\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-token\") pod \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\" (UID: \"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74\") " Nov 03 20:03:35.119529 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119446 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-socket" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.119529 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119480 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-data" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.119719 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119685 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-certs" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.119961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.119939 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istiod-ca-cert" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:35.120278 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.120256 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.120377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.120281 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.120377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.120295 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.120377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.120307 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.120377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.120321 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.122301 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.122272 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-kube-api-access-tcblz" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "kube-api-access-tcblz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:35.123259 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.123216 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-token" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:35.124929 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.124899 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-podinfo" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:03:35.125653 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.125627 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-envoy" pod "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" (UID: "2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:35.223259 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.223226 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcblz\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-kube-api-access-tcblz\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.223388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.223260 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.223388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.223274 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.223388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.223287 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:35.955553 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.955471 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" Nov 03 20:03:35.955553 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.955515 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk" event={"ID":"2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74","Type":"ContainerDied","Data":"56e1c2b4989ab519057384d90ccfd88820d68db990866be8c73fe72ad1b8472b"} Nov 03 20:03:35.956100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.955605 4314 scope.go:122] "RemoveContainer" containerID="ff20e67978f90292e541bc338fa9f854e6fcefa1a50a6e02fa090dd366a3cff0" Nov 03 20:03:35.957621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.957599 4314 generic.go:356] "Generic (PLEG): container finished" podID="b152554a-9b5f-4fae-9174-0662f4eda8c5" containerID="7632a5dfa87a9f3b1a8e4f420d2b84d6ad21a41c2092203fe155e2ba8b049e4d" exitCode=0 Nov 03 20:03:35.957780 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.957745 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" event={"ID":"b152554a-9b5f-4fae-9174-0662f4eda8c5","Type":"ContainerDied","Data":"7632a5dfa87a9f3b1a8e4f420d2b84d6ad21a41c2092203fe155e2ba8b049e4d"} Nov 03 20:03:36.001512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:35.997098 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk"] Nov 03 20:03:36.019604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:36.016077 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-79c47f-openshift-default-556b6749b5-hkzjk"] Nov 03 20:03:36.466175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:36.465542 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" path="/var/lib/kubelet/pods/2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74/volumes" Nov 03 20:03:37.169154 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.169121 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:37.266100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.266068 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b152554a-9b5f-4fae-9174-0662f4eda8c5-kube-api-access-g7jt7\" (UniqueName: \"kubernetes.io/projected/b152554a-9b5f-4fae-9174-0662f4eda8c5-kube-api-access-g7jt7\") pod \"b152554a-9b5f-4fae-9174-0662f4eda8c5\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " Nov 03 20:03:37.266278 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.266237 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-xks-ca\") pod \"b152554a-9b5f-4fae-9174-0662f4eda8c5\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " Nov 03 20:03:37.266334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.266298 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-openshift-ca\") pod \"b152554a-9b5f-4fae-9174-0662f4eda8c5\" (UID: \"b152554a-9b5f-4fae-9174-0662f4eda8c5\") " Nov 03 20:03:37.266779 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.266749 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-openshift-ca" pod "b152554a-9b5f-4fae-9174-0662f4eda8c5" (UID: "b152554a-9b5f-4fae-9174-0662f4eda8c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:37.267331 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.267307 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:37.268453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.268426 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b152554a-9b5f-4fae-9174-0662f4eda8c5-kube-api-access-g7jt7" pod "b152554a-9b5f-4fae-9174-0662f4eda8c5" (UID: "b152554a-9b5f-4fae-9174-0662f4eda8c5"). InnerVolumeSpecName "kube-api-access-g7jt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:37.268799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.268769 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-xks-ca" pod "b152554a-9b5f-4fae-9174-0662f4eda8c5" (UID: "b152554a-9b5f-4fae-9174-0662f4eda8c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:37.369332 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.369302 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jt7\" (UniqueName: \"kubernetes.io/projected/b152554a-9b5f-4fae-9174-0662f4eda8c5-kube-api-access-g7jt7\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:37.369428 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.369339 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b152554a-9b5f-4fae-9174-0662f4eda8c5-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:37.979423 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.979385 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" Nov 03 20:03:37.979423 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.979398 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk" event={"ID":"b152554a-9b5f-4fae-9174-0662f4eda8c5","Type":"ContainerDied","Data":"0a6fea7cecf09aa32a97c6461aebcd402348e34ca69d7f361ada392c2ec693ef"} Nov 03 20:03:37.979671 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:37.979467 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a6fea7cecf09aa32a97c6461aebcd402348e34ca69d7f361ada392c2ec693ef" Nov 03 20:03:46.174718 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.174568 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj"] Nov 03 20:03:46.176650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.176623 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b152554a-9b5f-4fae-9174-0662f4eda8c5" containerName="revoke-keys" Nov 03 20:03:46.176836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.176809 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b152554a-9b5f-4fae-9174-0662f4eda8c5" containerName="revoke-keys" Nov 03 20:03:46.176981 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.176967 4314 cpu_manager.go:414] "containerMap: removing container" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerName="istio-proxy" Nov 03 20:03:46.177084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.177073 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerName="istio-proxy" Nov 03 20:03:46.177421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.177405 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b152554a-9b5f-4fae-9174-0662f4eda8c5" containerName="revoke-keys" Nov 03 20:03:46.177534 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.177523 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb7ad2f-e8e4-45e2-8b1a-7cd98bcedb74" containerName="istio-proxy" Nov 03 20:03:46.301510 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.301472 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj"] Nov 03 20:03:46.307489 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.305882 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.314621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.314153 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:03:46.314621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.314223 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:03:46.424142 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.424111 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.424307 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.424289 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64nmz\" (UniqueName: \"kubernetes.io/projected/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-kube-api-access-64nmz\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.424455 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.424433 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.525399 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.525370 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64nmz\" (UniqueName: \"kubernetes.io/projected/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-kube-api-access-64nmz\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.525612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.525466 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.525667 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.525628 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.526333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.526300 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.528131 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.528111 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.533812 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.533791 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64nmz\" (UniqueName: \"kubernetes.io/projected/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-kube-api-access-64nmz\") pod \"maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.638718 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.638682 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:46.932888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:46.932559 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj"] Nov 03 20:03:46.933126 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:03:46.933097 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60e60ee_0a83_452d_b9ae_94bb5530cc3a.slice/crio-fc635da50614fd872d70da65e8649f13bd68f8f81ec0478eb689a30cb521bb15 WatchSource:0}: Error finding container fc635da50614fd872d70da65e8649f13bd68f8f81ec0478eb689a30cb521bb15: Status 404 returned error can't find the container with id fc635da50614fd872d70da65e8649f13bd68f8f81ec0478eb689a30cb521bb15 Nov 03 20:03:47.085156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:47.085107 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" event={"ID":"a60e60ee-0a83-452d-b9ae-94bb5530cc3a","Type":"ContainerStarted","Data":"fc635da50614fd872d70da65e8649f13bd68f8f81ec0478eb689a30cb521bb15"} Nov 03 20:03:48.097539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:48.097503 4314 generic.go:356] "Generic (PLEG): container finished" podID="a60e60ee-0a83-452d-b9ae-94bb5530cc3a" containerID="6d2b5544be0648d3bfc6f90aec459bdc54aa6a46ef0540d9817f8777501916c4" exitCode=0 Nov 03 20:03:48.098087 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:48.097634 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" event={"ID":"a60e60ee-0a83-452d-b9ae-94bb5530cc3a","Type":"ContainerDied","Data":"6d2b5544be0648d3bfc6f90aec459bdc54aa6a46ef0540d9817f8777501916c4"} Nov 03 20:03:48.709161 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:48.709090 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" containerID="cri-o://b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965" gracePeriod=45 Nov 03 20:03:49.032644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.032609 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 20:03:49.095458 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.095414 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-model-cache\") pod \"f468d654-86d4-4e80-a134-3c566229a9f5\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " Nov 03 20:03:49.095683 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.095497 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f468d654-86d4-4e80-a134-3c566229a9f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/f468d654-86d4-4e80-a134-3c566229a9f5-tls-certs\") pod \"f468d654-86d4-4e80-a134-3c566229a9f5\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " Nov 03 20:03:49.095683 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.095641 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-home\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-home\") pod \"f468d654-86d4-4e80-a134-3c566229a9f5\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " Nov 03 20:03:49.095827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.095685 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f468d654-86d4-4e80-a134-3c566229a9f5-kube-api-access-w9m9f\" (UniqueName: \"kubernetes.io/projected/f468d654-86d4-4e80-a134-3c566229a9f5-kube-api-access-w9m9f\") pod \"f468d654-86d4-4e80-a134-3c566229a9f5\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " Nov 03 20:03:49.095827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.095771 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-tmp-dir\") pod \"f468d654-86d4-4e80-a134-3c566229a9f5\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " Nov 03 20:03:49.095944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.095856 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-dshm\") pod \"f468d654-86d4-4e80-a134-3c566229a9f5\" (UID: \"f468d654-86d4-4e80-a134-3c566229a9f5\") " Nov 03 20:03:49.099722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.099687 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-dshm" pod "f468d654-86d4-4e80-a134-3c566229a9f5" (UID: "f468d654-86d4-4e80-a134-3c566229a9f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:49.100140 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.099915 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-model-cache" pod "f468d654-86d4-4e80-a134-3c566229a9f5" (UID: "f468d654-86d4-4e80-a134-3c566229a9f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:49.102373 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.102341 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f468d654-86d4-4e80-a134-3c566229a9f5-tls-certs" pod "f468d654-86d4-4e80-a134-3c566229a9f5" (UID: "f468d654-86d4-4e80-a134-3c566229a9f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:49.102567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.102545 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-home" pod "f468d654-86d4-4e80-a134-3c566229a9f5" (UID: "f468d654-86d4-4e80-a134-3c566229a9f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:49.108599 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.108553 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.108713 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.108603 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.108713 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.108619 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f468d654-86d4-4e80-a134-3c566229a9f5-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.108713 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.108635 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.109003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.108977 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-tmp-dir" pod "f468d654-86d4-4e80-a134-3c566229a9f5" (UID: "f468d654-86d4-4e80-a134-3c566229a9f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:49.110031 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.109994 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f468d654-86d4-4e80-a134-3c566229a9f5-kube-api-access-w9m9f" pod "f468d654-86d4-4e80-a134-3c566229a9f5" (UID: "f468d654-86d4-4e80-a134-3c566229a9f5"). InnerVolumeSpecName "kube-api-access-w9m9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:49.117630 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.117557 4314 generic.go:356] "Generic (PLEG): container finished" podID="f468d654-86d4-4e80-a134-3c566229a9f5" containerID="b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965" exitCode=0 Nov 03 20:03:49.117728 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.117654 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" Nov 03 20:03:49.117728 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.117659 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" event={"ID":"f468d654-86d4-4e80-a134-3c566229a9f5","Type":"ContainerDied","Data":"b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965"} Nov 03 20:03:49.117728 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.117709 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt" event={"ID":"f468d654-86d4-4e80-a134-3c566229a9f5","Type":"ContainerDied","Data":"3c67cb5adaa7f7afafbb01e4e3ec94866fc2de457637cec45d2b3015b5351088"} Nov 03 20:03:49.117843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.117732 4314 scope.go:122] "RemoveContainer" containerID="b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965" Nov 03 20:03:49.224651 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.222159 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9m9f\" (UniqueName: \"kubernetes.io/projected/f468d654-86d4-4e80-a134-3c566229a9f5-kube-api-access-w9m9f\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.224651 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.222228 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f468d654-86d4-4e80-a134-3c566229a9f5-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.300121 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.299831 4314 scope.go:122] "RemoveContainer" containerID="b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965" Nov 03 20:03:49.304763 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:49.304721 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965\": container with ID starting with b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965 not found: ID does not exist" containerID="b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965" Nov 03 20:03:49.304919 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.304774 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965"} err="failed to get container status \"b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965\": rpc error: code = NotFound desc = could not find container \"b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965\": container with ID starting with b0ce4c032bc7843648e1574cb31dfa2129b8c5c10747d7e5318f549974de7965 not found: ID does not exist" Nov 03 20:03:49.304919 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.304846 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt"] Nov 03 20:03:49.330952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.330917 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7a62d4/test-model-7a62d4-kserve-55f6d5c45f-kzqmt"] Nov 03 20:03:49.588552 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.588524 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:49.644605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.641813 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-xks-ca\") pod \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " Nov 03 20:03:49.644605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.641881 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-openshift-ca\") pod \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " Nov 03 20:03:49.644605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.642242 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-kube-api-access-64nmz\" (UniqueName: \"kubernetes.io/projected/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-kube-api-access-64nmz\") pod \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\" (UID: \"a60e60ee-0a83-452d-b9ae-94bb5530cc3a\") " Nov 03 20:03:49.644605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.642560 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-openshift-ca" pod "a60e60ee-0a83-452d-b9ae-94bb5530cc3a" (UID: "a60e60ee-0a83-452d-b9ae-94bb5530cc3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:49.655067 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.645571 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-kube-api-access-64nmz" pod "a60e60ee-0a83-452d-b9ae-94bb5530cc3a" (UID: "a60e60ee-0a83-452d-b9ae-94bb5530cc3a"). InnerVolumeSpecName "kube-api-access-64nmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:49.655067 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.646048 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64nmz\" (UniqueName: \"kubernetes.io/projected/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-kube-api-access-64nmz\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.655067 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.646081 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:49.655067 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.647575 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-xks-ca" pod "a60e60ee-0a83-452d-b9ae-94bb5530cc3a" (UID: "a60e60ee-0a83-452d-b9ae-94bb5530cc3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:03:49.747629 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:49.747568 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a60e60ee-0a83-452d-b9ae-94bb5530cc3a-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:50.129057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:50.129012 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" event={"ID":"a60e60ee-0a83-452d-b9ae-94bb5530cc3a","Type":"ContainerDied","Data":"fc635da50614fd872d70da65e8649f13bd68f8f81ec0478eb689a30cb521bb15"} Nov 03 20:03:50.129567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:50.129083 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc635da50614fd872d70da65e8649f13bd68f8f81ec0478eb689a30cb521bb15" Nov 03 20:03:50.129567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:50.129038 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj" Nov 03 20:03:50.550602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:50.523683 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" path="/var/lib/kubelet/pods/f468d654-86d4-4e80-a134-3c566229a9f5/volumes" Nov 03 20:03:50.620837 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:50.620801 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:03:51.992114 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:03:51.992073 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:03:53.601607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:53.597716 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:54.151609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.139802 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:54.345793 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.345727 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:54.527610 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.527555 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:03:54.528328 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.528297 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:03:54.547701 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.547656 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:54.600954 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.600908 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:54.601859 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.601834 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:03:54.732643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.725686 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:54.889641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:54.889504 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:55.530606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:55.529680 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:55.530606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:55.530129 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 03 20:03:55.530885 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:55.530751 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:03:55.530885 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:55.530810 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 20:03:55.724708 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:55.708145 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 20:03:55.739604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:55.725980 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:55.739604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:55.726320 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:03:56.045908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:56.045858 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:56.369667 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:56.351597 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:03:56.369997 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:56.353478 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:03:56.478606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:56.477503 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:03:56.676661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:56.520740 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 20:03:56.747739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:56.747694 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 20:03:56.879252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:56.807712 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.322625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.316361 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 20:03:57.359668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.354796 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.359958 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.359923 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded" Nov 03 20:03:57.405603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.371900 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:03:57.417724 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.407776 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 20:03:57.515606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.512772 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.668086 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.667974 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 20:03:57.669076 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.669052 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 20:03:57.821606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.820724 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:57.897245 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:57.897201 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 20:03:58.046509 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:58.046460 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:58.118725 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:58.073811 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 20:03:58.378659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:58.377957 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:03:58.378659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:58.378451 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:03:58.414605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:58.413904 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:03:58.704916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:58.704820 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 20:03:58.919691 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:58.899882 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:59.103871 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.100713 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 20:03:59.153405 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.153346 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:59.380430 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.379671 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:03:59.415865 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.414945 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:59.424242 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.423691 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:59.579479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.579447 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk"] Nov 03 20:03:59.720102 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.719664 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:03:59.927733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:03:59.927675 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:00.199608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.181664 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:00.328956 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.328910 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:04:00.389360 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.389318 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:00.415321 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.415274 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:00.437661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.430934 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded" Nov 03 20:04:00.539137 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.539105 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:04:00.632768 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.632647 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 20:04:00.647218 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.647180 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:04:00.737934 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.737888 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 20:04:00.751913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.747521 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:04:00.777801 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:00.777768 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 20:04:04.681530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:04.681486 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" podUID="7b72a3f1-73e6-471a-987f-fb30f6b73752" containerName="payload-pre-processing" containerID="cri-o://2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa" gracePeriod=25 Nov 03 20:04:05.362619 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:05.339715 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:04:05.362619 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:05.339784 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:05.583754 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:05.583715 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj"] Nov 03 20:04:05.591799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:05.591765 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a62d4-4xlqj"] Nov 03 20:04:06.431220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:06.431181 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60e60ee-0a83-452d-b9ae-94bb5530cc3a" path="/var/lib/kubelet/pods/a60e60ee-0a83-452d-b9ae-94bb5530cc3a/volumes" Nov 03 20:04:06.961792 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:06.961695 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:04:06.980201 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:06.961811 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:07.840075 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.840037 4314 generic.go:356] "Generic (PLEG): container finished" podID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerID="f11f1f8f5bfbbdacd77940fdd119b174e4de9db4215641c0862af2a5d7ec4ac2" exitCode=0 Nov 03 20:04:07.840569 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.840075 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" event={"ID":"85f0f2cb-9364-4969-aaca-da82fe5dd836","Type":"ContainerDied","Data":"f11f1f8f5bfbbdacd77940fdd119b174e4de9db4215641c0862af2a5d7ec4ac2"} Nov 03 20:04:07.840704 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.840673 4314 scope.go:122] "RemoveContainer" containerID="f11f1f8f5bfbbdacd77940fdd119b174e4de9db4215641c0862af2a5d7ec4ac2" Nov 03 20:04:07.843318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.843283 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.843420 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.843361 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.843420 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.843394 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.849056 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.849032 4314 generic.go:356] "Generic (PLEG): container finished" podID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerID="7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a" exitCode=0 Nov 03 20:04:07.849180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.849131 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" event={"ID":"1fedb898-5b21-48a4-bf0d-3f603e9a717b","Type":"ContainerDied","Data":"7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a"} Nov 03 20:04:07.849832 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.849810 4314 scope.go:122] "RemoveContainer" containerID="7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a" Nov 03 20:04:07.857054 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.857015 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.857170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.857105 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.857170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.857151 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.859414 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.859388 4314 generic.go:356] "Generic (PLEG): container finished" podID="a7076438-f898-42dc-80dc-9af5392701a5" containerID="5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b" exitCode=0 Nov 03 20:04:07.859577 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.859521 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" event={"ID":"a7076438-f898-42dc-80dc-9af5392701a5","Type":"ContainerDied","Data":"5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b"} Nov 03 20:04:07.861419 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.859947 4314 scope.go:122] "RemoveContainer" containerID="5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b" Nov 03 20:04:07.881727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.876917 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.881727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.876999 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:07.881727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:07.877044 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:08.872370 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:08.872327 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" event={"ID":"85f0f2cb-9364-4969-aaca-da82fe5dd836","Type":"ContainerStarted","Data":"3e3634a7c2862d5f4c51217ebdc94af55fc39dbed3d4845402abe9407bcd1f6c"} Nov 03 20:04:08.875257 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:08.875224 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" event={"ID":"1fedb898-5b21-48a4-bf0d-3f603e9a717b","Type":"ContainerStarted","Data":"afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308"} Nov 03 20:04:08.877726 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:08.877698 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" event={"ID":"a7076438-f898-42dc-80dc-9af5392701a5","Type":"ContainerStarted","Data":"d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030"} Nov 03 20:04:09.231665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.231132 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:04:09.239300 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.232282 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:09.239300 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.232344 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:09.408604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.407744 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:04:09.408604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.408398 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:09.408604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.408471 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:09.610826 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.610792 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 20:04:09.611163 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.611141 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:09.611309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:09.611287 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:10.071607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.070820 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:04:10.227074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.227037 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:10.258220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.251037 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:10.613638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.613424 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:10.613638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.613486 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:10.634609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.627737 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:10.634609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.627794 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:10.841602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:10.833115 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:11.252607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.250890 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:11.252607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.250949 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:11.409616 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.408544 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:11.409616 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.408621 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:11.619382 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.616501 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:11.619382 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.616576 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:11.806607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.795654 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:11.812607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.811671 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:11.843601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.842622 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:11.843601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.843202 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:04:11.982501 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:11.957119 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:04:12.284735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.284696 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:12.303702 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.303660 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:12.415408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.386932 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:04:12.415408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.409010 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:12.415408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.409066 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:12.490635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.486641 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:12.817604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.809685 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:12.936609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.897710 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded" Nov 03 20:04:12.992609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:12.988709 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:13.013012 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.012957 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:13.109170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.109082 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.109457 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.109416 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:13.156676 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.155498 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:13.157878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.157154 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 20:04:13.301996 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.295333 4314 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" secret="" err="secret \"e2e-inf-a-7a62d4-openshift-default-dockercfg-vb879\" not found" Nov 03 20:04:13.447409 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.388548 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.507612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.497459 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:13.507612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.453950 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 20:04:13.536982 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.398712 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:13.537364 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.537337 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:04:13.537628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.537601 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:13.742663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.691430 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 20:04:13.742663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.691529 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:04:13.920650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.916943 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.920650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.917025 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:13.920650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.917718 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:13.920650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:13.917777 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:14.075499 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.075402 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7"] Nov 03 20:04:14.105276 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.105228 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:14.110679 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.110638 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 20:04:14.111612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.111563 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": context deadline exceeded" Nov 03 20:04:14.112994 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.112971 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 20:04:14.125603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.123907 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:04:14.125603 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.124071 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:04:14.218774 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.218728 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:14.232284 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.232256 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 20:04:14.279845 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.279809 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.280098 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.280071 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:14.379691 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.379595 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" podUID="7b72a3f1-73e6-471a-987f-fb30f6b73752" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:04:14.432844 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.432805 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.618560 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.618510 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:14.620421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.589282 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:04:14.620670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.437920 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:14.652937 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.650488 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:14.652937 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.650556 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:14.673605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.660160 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l"] Nov 03 20:04:14.673605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.661402 4314 cpu_manager.go:414] "containerMap: removing container" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" Nov 03 20:04:14.673605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.661424 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" Nov 03 20:04:14.673605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.661464 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a60e60ee-0a83-452d-b9ae-94bb5530cc3a" containerName="revoke-keys" Nov 03 20:04:14.673605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.661473 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60e60ee-0a83-452d-b9ae-94bb5530cc3a" containerName="revoke-keys" Nov 03 20:04:14.673605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.661741 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f468d654-86d4-4e80-a134-3c566229a9f5" containerName="main" Nov 03 20:04:14.673605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.661773 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60e60ee-0a83-452d-b9ae-94bb5530cc3a" containerName="revoke-keys" Nov 03 20:04:14.718675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.694896 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:14.752842 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:14.675797 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.138725 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.138687 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l"] Nov 03 20:04:15.138961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.138823 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.140070 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.140029 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.141106 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.141064 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.141394 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.141368 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.144912 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.144891 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:04:15.145108 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.145092 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:15.261037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.260996 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:15.261279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.261252 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:15.281006 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.264646 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" containerID="cri-o://b70707b1082ef4b21aff627053f4896b50eba458f547b09888b0222db90a69ff" gracePeriod=30 Nov 03 20:04:15.287942 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.277803 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.359606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.334737 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.359999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.359964 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5hd\" (UniqueName: \"kubernetes.io/projected/5548e58d-360e-4b10-a177-166cbe7b7924-kube-api-access-nj5hd\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.360427 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.360401 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.382906 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.382860 4314 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.410783 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.410693 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:15.411041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.411012 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:15.484270 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.483945 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.489101 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.489060 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5hd\" (UniqueName: \"kubernetes.io/projected/5548e58d-360e-4b10-a177-166cbe7b7924-kube-api-access-nj5hd\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.501894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.489874 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.503517 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.491364 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.555008 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.554938 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5hd\" (UniqueName: \"kubernetes.io/projected/5548e58d-360e-4b10-a177-166cbe7b7924-kube-api-access-nj5hd\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.624661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.601666 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:15.624661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.610981 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:15.624661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.611035 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:15.624661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.612044 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d12868-xct7l\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.645607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.644710 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:15.725709 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.725668 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:04:15.766605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:15.753004 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:16.176631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.150681 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 20:04:16.176631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.151287 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 20:04:16.176631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.151743 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:16.331108 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.285145 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:04:16.331108 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.285206 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 20:04:16.349628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.349562 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:04:16.422625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.409092 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:04:16.422625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.409167 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:04:16.580513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.572877 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 20:04:16.580513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.573194 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 20:04:16.580513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.573233 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 20:04:16.580513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.573265 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:04:16.580513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.573296 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:04:16.586635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.585805 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 20:04:16.627072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.626793 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:04:16.627072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.627028 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 20:04:16.776810 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:16.776777 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l"] Nov 03 20:04:17.229804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.229764 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:04:17.231049 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.231017 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:04:17.415176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.415145 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:04:17.415608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.415573 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:04:17.416430 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.416410 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:04:17.561939 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.561830 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" event={"ID":"5548e58d-360e-4b10-a177-166cbe7b7924","Type":"ContainerStarted","Data":"8f10fda60f6eea1b4c87e3adaf6b415a0be6a3fea27a666c3ee9b69be84d5991"} Nov 03 20:04:17.562394 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.561937 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" event={"ID":"5548e58d-360e-4b10-a177-166cbe7b7924","Type":"ContainerStarted","Data":"b322f19de3a90e26fe164b727d22050c4b64f9e016a3fb72ec0e30697f5f006a"} Nov 03 20:04:17.589604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.586188 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" podStartSLOduration=4.586170336 podStartE2EDuration="4.586170336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:17.584663556 +0000 UTC m=+1965.289502897" watchObservedRunningTime="2026-11-03 20:04:17.586170336 +0000 UTC m=+1965.291009676" Nov 03 20:04:17.615839 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.615810 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 20:04:17.616179 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.616159 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 20:04:17.616778 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:17.616759 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 20:04:18.262156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:18.262120 4314 patch_prober.go:29] interesting pod/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:04:18.262333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:18.262198 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:04:18.571971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:18.571881 4314 generic.go:356] "Generic (PLEG): container finished" podID="5548e58d-360e-4b10-a177-166cbe7b7924" containerID="8f10fda60f6eea1b4c87e3adaf6b415a0be6a3fea27a666c3ee9b69be84d5991" exitCode=0 Nov 03 20:04:18.572413 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:18.571973 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" event={"ID":"5548e58d-360e-4b10-a177-166cbe7b7924","Type":"ContainerDied","Data":"8f10fda60f6eea1b4c87e3adaf6b415a0be6a3fea27a666c3ee9b69be84d5991"} Nov 03 20:04:19.771848 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.771822 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:19.828371 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.828337 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-openshift-ca\") pod \"5548e58d-360e-4b10-a177-166cbe7b7924\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " Nov 03 20:04:19.828570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.828505 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-xks-ca\") pod \"5548e58d-360e-4b10-a177-166cbe7b7924\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " Nov 03 20:04:19.828570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.828535 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5548e58d-360e-4b10-a177-166cbe7b7924-kube-api-access-nj5hd\" (UniqueName: \"kubernetes.io/projected/5548e58d-360e-4b10-a177-166cbe7b7924-kube-api-access-nj5hd\") pod \"5548e58d-360e-4b10-a177-166cbe7b7924\" (UID: \"5548e58d-360e-4b10-a177-166cbe7b7924\") " Nov 03 20:04:19.830058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.830001 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-openshift-ca" pod "5548e58d-360e-4b10-a177-166cbe7b7924" (UID: "5548e58d-360e-4b10-a177-166cbe7b7924"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:19.832033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.831965 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-xks-ca" pod "5548e58d-360e-4b10-a177-166cbe7b7924" (UID: "5548e58d-360e-4b10-a177-166cbe7b7924"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:19.832401 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.832374 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5548e58d-360e-4b10-a177-166cbe7b7924-kube-api-access-nj5hd" pod "5548e58d-360e-4b10-a177-166cbe7b7924" (UID: "5548e58d-360e-4b10-a177-166cbe7b7924"). InnerVolumeSpecName "kube-api-access-nj5hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:19.931173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.931140 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:19.931173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.931173 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548e58d-360e-4b10-a177-166cbe7b7924-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:19.931367 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:19.931184 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5hd\" (UniqueName: \"kubernetes.io/projected/5548e58d-360e-4b10-a177-166cbe7b7924-kube-api-access-nj5hd\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.595658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.595573 4314 generic.go:356] "Generic (PLEG): container finished" podID="fa2d0438-5595-42d4-821e-5618f05382fd" containerID="b70707b1082ef4b21aff627053f4896b50eba458f547b09888b0222db90a69ff" exitCode=0 Nov 03 20:04:20.595875 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.595639 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" event={"ID":"fa2d0438-5595-42d4-821e-5618f05382fd","Type":"ContainerDied","Data":"b70707b1082ef4b21aff627053f4896b50eba458f547b09888b0222db90a69ff"} Nov 03 20:04:20.596000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.595916 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" event={"ID":"fa2d0438-5595-42d4-821e-5618f05382fd","Type":"ContainerDied","Data":"40b6cd60b61036c4b6f56fc4cf450b0f78cb388f4ffb45696a9e7f0cc90b399f"} Nov 03 20:04:20.596000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.595947 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b6cd60b61036c4b6f56fc4cf450b0f78cb388f4ffb45696a9e7f0cc90b399f" Nov 03 20:04:20.596000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.595965 4314 scope.go:122] "RemoveContainer" containerID="14de153be85fc5e6443a99cc5621f5e19a9867e9f647f414a2b55368494e3463" Nov 03 20:04:20.597886 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.597866 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 20:04:20.597977 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.597895 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" event={"ID":"5548e58d-360e-4b10-a177-166cbe7b7924","Type":"ContainerDied","Data":"b322f19de3a90e26fe164b727d22050c4b64f9e016a3fb72ec0e30697f5f006a"} Nov 03 20:04:20.597977 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.597928 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l" Nov 03 20:04:20.597977 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.597941 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b322f19de3a90e26fe164b727d22050c4b64f9e016a3fb72ec0e30697f5f006a" Nov 03 20:04:20.642750 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.642547 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2d0438-5595-42d4-821e-5618f05382fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2d0438-5595-42d4-821e-5618f05382fd-istiod-ca-cert\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.642750 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.642700 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-socket\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.643000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.642756 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-certs\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.643000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.642852 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-kube-api-access-q5z2s\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-kube-api-access-q5z2s\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.643000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.642881 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-envoy\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.643000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.642986 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa2d0438-5595-42d4-821e-5618f05382fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2d0438-5595-42d4-821e-5618f05382fd-istio-podinfo\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.643321 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.643028 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-data\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.643321 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.643076 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-credential-socket\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.643321 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.643114 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-istio-token\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-istio-token\") pod \"fa2d0438-5595-42d4-821e-5618f05382fd\" (UID: \"fa2d0438-5595-42d4-821e-5618f05382fd\") " Nov 03 20:04:20.646184 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.646045 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-certs" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:20.646378 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.646323 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-data" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:20.646822 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.646796 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-socket" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:20.646822 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.646807 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa2d0438-5595-42d4-821e-5618f05382fd-istio-podinfo" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:20.647005 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.646896 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-istio-token" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:20.647068 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.647047 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-credential-socket" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:20.647190 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.647131 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2d0438-5595-42d4-821e-5618f05382fd-istiod-ca-cert" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:20.649659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.648631 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-kube-api-access-q5z2s" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "kube-api-access-q5z2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:20.649659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.648969 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-envoy" pod "fa2d0438-5595-42d4-821e-5618f05382fd" (UID: "fa2d0438-5595-42d4-821e-5618f05382fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:20.745662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745628 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745655 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745665 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5z2s\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-kube-api-access-q5z2s\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745869 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745676 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745869 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745686 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2d0438-5595-42d4-821e-5618f05382fd-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745869 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745694 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745869 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745703 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2d0438-5595-42d4-821e-5618f05382fd-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745869 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745712 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2d0438-5595-42d4-821e-5618f05382fd-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:20.745869 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:20.745720 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2d0438-5595-42d4-821e-5618f05382fd-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:21.608237 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:21.607944 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7" Nov 03 20:04:21.647330 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:21.646515 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7"] Nov 03 20:04:21.653643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:21.653613 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7a62d4-openshift-default-745855d459-qjhv7"] Nov 03 20:04:22.068739 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:22.068401 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:04:22.434254 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:22.434173 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" path="/var/lib/kubelet/pods/fa2d0438-5595-42d4-821e-5618f05382fd/volumes" Nov 03 20:04:23.745814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:23.745781 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm"] Nov 03 20:04:23.766303 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:23.766271 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx"] Nov 03 20:04:24.078267 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:24.077694 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" podUID="7b72a3f1-73e6-471a-987f-fb30f6b73752" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:04:24.791093 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:24.791061 4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b72a3f1_73e6_471a_987f_fb30f6b73752.slice/crio-conmon-2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:04:25.069132 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.069104 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 20:04:25.184063 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.184033 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b72a3f1-73e6-471a-987f-fb30f6b73752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b72a3f1-73e6-471a-987f-fb30f6b73752-plugins-config-volume\") pod \"7b72a3f1-73e6-471a-987f-fb30f6b73752\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " Nov 03 20:04:25.184384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.184360 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b72a3f1-73e6-471a-987f-fb30f6b73752-kube-api-access-tv6ns\" (UniqueName: \"kubernetes.io/projected/7b72a3f1-73e6-471a-987f-fb30f6b73752-kube-api-access-tv6ns\") pod \"7b72a3f1-73e6-471a-987f-fb30f6b73752\" (UID: \"7b72a3f1-73e6-471a-987f-fb30f6b73752\") " Nov 03 20:04:25.184513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.184365 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b72a3f1-73e6-471a-987f-fb30f6b73752-plugins-config-volume" pod "7b72a3f1-73e6-471a-987f-fb30f6b73752" (UID: "7b72a3f1-73e6-471a-987f-fb30f6b73752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:25.184911 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.184890 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b72a3f1-73e6-471a-987f-fb30f6b73752-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:25.186635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.186609 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b72a3f1-73e6-471a-987f-fb30f6b73752-kube-api-access-tv6ns" pod "7b72a3f1-73e6-471a-987f-fb30f6b73752" (UID: "7b72a3f1-73e6-471a-987f-fb30f6b73752"). InnerVolumeSpecName "kube-api-access-tv6ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:25.286242 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.286205 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6ns\" (UniqueName: \"kubernetes.io/projected/7b72a3f1-73e6-471a-987f-fb30f6b73752-kube-api-access-tv6ns\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:25.761146 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.761109 4314 generic.go:356] "Generic (PLEG): container finished" podID="7b72a3f1-73e6-471a-987f-fb30f6b73752" containerID="2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa" exitCode=0 Nov 03 20:04:25.761318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.761166 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" Nov 03 20:04:25.761318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.761197 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" event={"ID":"7b72a3f1-73e6-471a-987f-fb30f6b73752","Type":"ContainerDied","Data":"2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa"} Nov 03 20:04:25.761318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.761262 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk" event={"ID":"7b72a3f1-73e6-471a-987f-fb30f6b73752","Type":"ContainerDied","Data":"aa68efc1078081c9d8e29fc0ed5aa757dbcf0d90ee7572126574555223bc97db"} Nov 03 20:04:25.761318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.761294 4314 scope.go:122] "RemoveContainer" containerID="2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa" Nov 03 20:04:25.780138 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.780115 4314 scope.go:122] "RemoveContainer" containerID="2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa" Nov 03 20:04:25.780775 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:25.780746 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa\": container with ID starting with 2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa not found: ID does not exist" containerID="2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa" Nov 03 20:04:25.780895 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.780790 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa"} err="failed to get container status \"2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa\": rpc error: code = NotFound desc = could not find container \"2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa\": container with ID starting with 2e0219ea08a7224554434c999d5370e3dfb084d00f146aa90d270ee928ccefaa not found: ID does not exist" Nov 03 20:04:25.798290 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.798263 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk"] Nov 03 20:04:25.802499 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:25.802475 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-11c1a8-78df5d9ff9-chjdk"] Nov 03 20:04:26.431459 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:26.431427 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b72a3f1-73e6-471a-987f-fb30f6b73752" path="/var/lib/kubelet/pods/7b72a3f1-73e6-471a-987f-fb30f6b73752/volumes" Nov 03 20:04:28.746396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:28.746345 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" podUID="feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" containerName="payload-processing" containerID="cri-o://732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7" gracePeriod=25 Nov 03 20:04:28.766610 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:28.766559 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" podUID="b92ce758-b31d-46a2-990f-4f222c049a1f" containerName="payload-pre-processing" containerID="cri-o://c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0" gracePeriod=25 Nov 03 20:04:29.048541 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.048511 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 20:04:29.147942 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.147895 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-plugins-config-volume\") pod \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " Nov 03 20:04:29.148199 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.148177 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-kube-api-access-knqcv\" (UniqueName: \"kubernetes.io/projected/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-kube-api-access-knqcv\") pod \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\" (UID: \"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b\") " Nov 03 20:04:29.148317 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.148290 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-plugins-config-volume" pod "feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" (UID: "feb52a4a-1ec1-4a25-9ee6-c29d69884d5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:29.148945 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.148918 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:29.150778 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.150741 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-kube-api-access-knqcv" pod "feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" (UID: "feb52a4a-1ec1-4a25-9ee6-c29d69884d5b"). InnerVolumeSpecName "kube-api-access-knqcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:29.250529 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.250496 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqcv\" (UniqueName: \"kubernetes.io/projected/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b-kube-api-access-knqcv\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:29.522764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.522726 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp"] Nov 03 20:04:29.523288 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.523254 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" containerID="cri-o://50fb4ba4b1c76a6cad6f1222e6e9d73862813b0ce39db1cef4e0a1f835aeca23" gracePeriod=30 Nov 03 20:04:29.594780 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.594716 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9"] Nov 03 20:04:29.595244 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.595212 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" containerID="cri-o://1b930b64d1860f96ef932512107a8fa4a9740ba52fb3fcd5896c4d8609741932" gracePeriod=30 Nov 03 20:04:29.603046 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.603010 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq"] Nov 03 20:04:29.603714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.603574 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" containerID="cri-o://e0871c6f718aa1ed48432c7d8442606c730187cd81269a9e217bf27d2d53f741" gracePeriod=30 Nov 03 20:04:29.620017 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.619715 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp"] Nov 03 20:04:29.621009 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.619946 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" containerID="cri-o://9f3633a327e4aadacde7aea34daa2cb7c66d41a7006445e9397d5720e9b25513" gracePeriod=30 Nov 03 20:04:29.896043 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.895960 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh"] Nov 03 20:04:29.897646 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.897622 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" Nov 03 20:04:29.897788 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.897775 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" Nov 03 20:04:29.897888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.897878 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" Nov 03 20:04:29.897962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.897953 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" Nov 03 20:04:29.898041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898032 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7b72a3f1-73e6-471a-987f-fb30f6b73752" containerName="payload-pre-processing" Nov 03 20:04:29.898172 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898162 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b72a3f1-73e6-471a-987f-fb30f6b73752" containerName="payload-pre-processing" Nov 03 20:04:29.898245 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898237 4314 cpu_manager.go:414] "containerMap: removing container" podUID="feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" containerName="payload-processing" Nov 03 20:04:29.898316 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898307 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" containerName="payload-processing" Nov 03 20:04:29.898405 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898395 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5548e58d-360e-4b10-a177-166cbe7b7924" containerName="revoke-keys" Nov 03 20:04:29.898476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898467 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5548e58d-360e-4b10-a177-166cbe7b7924" containerName="revoke-keys" Nov 03 20:04:29.898834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898819 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" Nov 03 20:04:29.898948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.898938 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2d0438-5595-42d4-821e-5618f05382fd" containerName="istio-proxy" Nov 03 20:04:29.905130 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.905110 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b72a3f1-73e6-471a-987f-fb30f6b73752" containerName="payload-pre-processing" Nov 03 20:04:29.905263 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.905252 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" containerName="payload-processing" Nov 03 20:04:29.905349 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.905340 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5548e58d-360e-4b10-a177-166cbe7b7924" containerName="revoke-keys" Nov 03 20:04:29.917598 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.914728 4314 generic.go:356] "Generic (PLEG): container finished" podID="feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" containerID="732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7" exitCode=0 Nov 03 20:04:29.917598 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.915078 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" Nov 03 20:04:29.931849 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.931826 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh"] Nov 03 20:04:29.932144 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.932123 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" event={"ID":"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b","Type":"ContainerDied","Data":"732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7"} Nov 03 20:04:29.932306 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.932289 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm" event={"ID":"feb52a4a-1ec1-4a25-9ee6-c29d69884d5b","Type":"ContainerDied","Data":"6630cd1a02ae6a78ddc47d5cc17ada91d547367ad075c5b45a0c7524d7a7120a"} Nov 03 20:04:29.946612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.932070 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:29.950216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:29.932402 4314 scope.go:122] "RemoveContainer" containerID="732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7" Nov 03 20:04:30.007444 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.007189 4314 scope.go:122] "RemoveContainer" containerID="732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7" Nov 03 20:04:30.012692 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:30.009118 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7\": container with ID starting with 732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7 not found: ID does not exist" containerID="732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7" Nov 03 20:04:30.012692 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.009160 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7"} err="failed to get container status \"732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7\": rpc error: code = NotFound desc = could not find container \"732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7\": container with ID starting with 732b179498264894fd9a84bf11ab0574fed2dd70903ecc3f02c38802475070e7 not found: ID does not exist" Nov 03 20:04:30.020639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.019864 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:30.020639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.019997 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:04:30.031671 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.031647 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm"] Nov 03 20:04:30.034702 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.034664 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d12868-7bf6f9d95c-svxnm"] Nov 03 20:04:30.127972 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.127930 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcm4h\" (UniqueName: \"kubernetes.io/projected/6de80759-da72-4d5e-bca1-e30fde86e990-kube-api-access-zcm4h\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.128162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.128050 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.128162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.128077 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.229530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.229448 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.229530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.229495 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.229793 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.229779 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcm4h\" (UniqueName: \"kubernetes.io/projected/6de80759-da72-4d5e-bca1-e30fde86e990-kube-api-access-zcm4h\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.230263 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.230240 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.231882 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.231862 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.242488 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.242460 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcm4h\" (UniqueName: \"kubernetes.io/projected/6de80759-da72-4d5e-bca1-e30fde86e990-kube-api-access-zcm4h\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.339429 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.339397 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:30.440994 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.440962 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb52a4a-1ec1-4a25-9ee6-c29d69884d5b" path="/var/lib/kubelet/pods/feb52a4a-1ec1-4a25-9ee6-c29d69884d5b/volumes" Nov 03 20:04:30.639633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.636908 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh"] Nov 03 20:04:30.639633 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:04:30.638799 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de80759_da72_4d5e_bca1_e30fde86e990.slice/crio-5616db7a0397b464312e6dd12282ad6b51cd009566f1495b63b86af2b38063e0 WatchSource:0}: Error finding container 5616db7a0397b464312e6dd12282ad6b51cd009566f1495b63b86af2b38063e0: Status 404 returned error can't find the container with id 5616db7a0397b464312e6dd12282ad6b51cd009566f1495b63b86af2b38063e0 Nov 03 20:04:30.925916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:30.925840 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" event={"ID":"6de80759-da72-4d5e-bca1-e30fde86e990","Type":"ContainerStarted","Data":"5616db7a0397b464312e6dd12282ad6b51cd009566f1495b63b86af2b38063e0"} Nov 03 20:04:31.935313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:31.935235 4314 generic.go:356] "Generic (PLEG): container finished" podID="6de80759-da72-4d5e-bca1-e30fde86e990" containerID="cfae0006e342d85a1c43bae7b58c5ba2d8845b07110ae5a28f729826b0cd9682" exitCode=0 Nov 03 20:04:31.935313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:31.935307 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" event={"ID":"6de80759-da72-4d5e-bca1-e30fde86e990","Type":"ContainerDied","Data":"cfae0006e342d85a1c43bae7b58c5ba2d8845b07110ae5a28f729826b0cd9682"} Nov 03 20:04:32.323463 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:32.323424 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:04:32.323654 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:32.323478 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:04:32.928793 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:32.928757 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" podUID="b92ce758-b31d-46a2-990f-4f222c049a1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:04:33.122555 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.122523 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:33.181132 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.181045 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de80759-da72-4d5e-bca1-e30fde86e990-kube-api-access-zcm4h\" (UniqueName: \"kubernetes.io/projected/6de80759-da72-4d5e-bca1-e30fde86e990-kube-api-access-zcm4h\") pod \"6de80759-da72-4d5e-bca1-e30fde86e990\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " Nov 03 20:04:33.181311 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.181152 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-xks-ca\") pod \"6de80759-da72-4d5e-bca1-e30fde86e990\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " Nov 03 20:04:33.181402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.181386 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-openshift-ca\") pod \"6de80759-da72-4d5e-bca1-e30fde86e990\" (UID: \"6de80759-da72-4d5e-bca1-e30fde86e990\") " Nov 03 20:04:33.181927 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.181779 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-openshift-ca" pod "6de80759-da72-4d5e-bca1-e30fde86e990" (UID: "6de80759-da72-4d5e-bca1-e30fde86e990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:33.182197 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.182171 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:33.183431 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.183403 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de80759-da72-4d5e-bca1-e30fde86e990-kube-api-access-zcm4h" pod "6de80759-da72-4d5e-bca1-e30fde86e990" (UID: "6de80759-da72-4d5e-bca1-e30fde86e990"). InnerVolumeSpecName "kube-api-access-zcm4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:33.183626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.183600 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-xks-ca" pod "6de80759-da72-4d5e-bca1-e30fde86e990" (UID: "6de80759-da72-4d5e-bca1-e30fde86e990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:33.284186 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.284157 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcm4h\" (UniqueName: \"kubernetes.io/projected/6de80759-da72-4d5e-bca1-e30fde86e990-kube-api-access-zcm4h\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:33.284186 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:33.284184 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de80759-da72-4d5e-bca1-e30fde86e990-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:34.007627 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.007569 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" event={"ID":"6de80759-da72-4d5e-bca1-e30fde86e990","Type":"ContainerDied","Data":"5616db7a0397b464312e6dd12282ad6b51cd009566f1495b63b86af2b38063e0"} Nov 03 20:04:34.007791 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.007657 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5616db7a0397b464312e6dd12282ad6b51cd009566f1495b63b86af2b38063e0" Nov 03 20:04:34.007791 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.007731 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh" Nov 03 20:04:34.372954 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.372861 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz"] Nov 03 20:04:34.374092 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.374063 4314 cpu_manager.go:414] "containerMap: removing container" podUID="6de80759-da72-4d5e-bca1-e30fde86e990" containerName="revoke-keys" Nov 03 20:04:34.374256 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.374238 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de80759-da72-4d5e-bca1-e30fde86e990" containerName="revoke-keys" Nov 03 20:04:34.374568 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.374548 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de80759-da72-4d5e-bca1-e30fde86e990" containerName="revoke-keys" Nov 03 20:04:34.379455 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.379427 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.416830 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.416796 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz"] Nov 03 20:04:34.418287 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418252 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418416 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418371 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418497 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418467 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418614 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418535 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418614 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418570 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418720 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418632 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418720 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418692 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418724 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tb8x\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-kube-api-access-5tb8x\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.418823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.418766 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.520613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.520548 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.520613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.520612 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.520840 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.520634 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tb8x\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-kube-api-access-5tb8x\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.520840 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.520659 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.520840 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.520829 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.520995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.520927 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521053 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521145 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521075 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521145 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521103 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521140 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521150 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521271 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521438 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521415 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.521493 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.521413 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.523352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.523323 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.523617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.523562 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.528737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.528714 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tb8x\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-kube-api-access-5tb8x\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.528827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.528718 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.717939 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.717907 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:34.823631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.819316 4314 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 20:04:34.823631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:34.819390 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 20:04:35.037287 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.037250 4314 generic.go:356] "Generic (PLEG): container finished" podID="db981728-088c-4efd-a9b8-6cdee6b42866" containerID="9f3633a327e4aadacde7aea34daa2cb7c66d41a7006445e9397d5720e9b25513" exitCode=0 Nov 03 20:04:35.037476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.037345 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" event={"ID":"db981728-088c-4efd-a9b8-6cdee6b42866","Type":"ContainerDied","Data":"9f3633a327e4aadacde7aea34daa2cb7c66d41a7006445e9397d5720e9b25513"} Nov 03 20:04:35.039637 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.039603 4314 generic.go:356] "Generic (PLEG): container finished" podID="40ba5076-0c89-43ed-a408-0005716f7a87" containerID="e0871c6f718aa1ed48432c7d8442606c730187cd81269a9e217bf27d2d53f741" exitCode=0 Nov 03 20:04:35.039783 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.039677 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" event={"ID":"40ba5076-0c89-43ed-a408-0005716f7a87","Type":"ContainerDied","Data":"e0871c6f718aa1ed48432c7d8442606c730187cd81269a9e217bf27d2d53f741"} Nov 03 20:04:35.041941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.041915 4314 generic.go:356] "Generic (PLEG): container finished" podID="02179854-1285-4182-aca7-1e083832067a" containerID="1b930b64d1860f96ef932512107a8fa4a9740ba52fb3fcd5896c4d8609741932" exitCode=0 Nov 03 20:04:35.042399 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.042080 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" event={"ID":"02179854-1285-4182-aca7-1e083832067a","Type":"ContainerDied","Data":"1b930b64d1860f96ef932512107a8fa4a9740ba52fb3fcd5896c4d8609741932"} Nov 03 20:04:35.045310 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.045274 4314 generic.go:356] "Generic (PLEG): container finished" podID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerID="50fb4ba4b1c76a6cad6f1222e6e9d73862813b0ce39db1cef4e0a1f835aeca23" exitCode=0 Nov 03 20:04:35.045745 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.045542 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" event={"ID":"3c349a0a-69ad-4f14-ba0d-292c4a009b17","Type":"ContainerDied","Data":"50fb4ba4b1c76a6cad6f1222e6e9d73862813b0ce39db1cef4e0a1f835aeca23"} Nov 03 20:04:35.045745 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.045618 4314 scope.go:122] "RemoveContainer" containerID="1c30e50cbf659931eb96c80e8fbfee4c08d3e6197971195b936edc7c2940643c" Nov 03 20:04:35.138638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.133830 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:04:35.138638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.137820 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 20:04:35.141874 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.141843 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 20:04:35.204555 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.204452 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:04:35.241831 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.241796 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-istio-token\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-istio-token\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.242006 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.241892 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-kube-api-access-kgw5z\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-kube-api-access-kgw5z\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.242006 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.241935 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40ba5076-0c89-43ed-a408-0005716f7a87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba5076-0c89-43ed-a408-0005716f7a87-istio-podinfo\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.242006 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.241995 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02179854-1285-4182-aca7-1e083832067a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02179854-1285-4182-aca7-1e083832067a-istio-podinfo\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.242194 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242037 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-envoy\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.242287 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242202 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-data\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.242287 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242246 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-socket\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.242378 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242284 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ba5076-0c89-43ed-a408-0005716f7a87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba5076-0c89-43ed-a408-0005716f7a87-istiod-ca-cert\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.242378 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242331 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-istio-token\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-istio-token\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.242378 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242368 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db981728-088c-4efd-a9b8-6cdee6b42866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db981728-088c-4efd-a9b8-6cdee6b42866-istio-podinfo\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.242542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242415 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-data\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.242542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242448 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-data\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.242542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242476 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-socket\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.242542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242507 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-envoy\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.242773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242551 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02179854-1285-4182-aca7-1e083832067a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02179854-1285-4182-aca7-1e083832067a-istiod-ca-cert\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.242773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242608 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-data\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.242880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242799 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-istio-token\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-istio-token\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.242880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242843 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-certs\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.243000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242868 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-data" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.243000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242886 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-credential-socket\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.243000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242923 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istiod-ca-cert\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.243000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.242979 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-socket\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.243215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243021 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db981728-088c-4efd-a9b8-6cdee6b42866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db981728-088c-4efd-a9b8-6cdee6b42866-istiod-ca-cert\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.243215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243049 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-certs\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.243215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243096 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-kube-api-access-5x56b\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-kube-api-access-5x56b\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.243215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243128 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-credential-socket\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.243215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243159 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-kube-api-access-5p4f7\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-kube-api-access-5p4f7\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.243215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243205 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-envoy\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.243530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243266 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-kube-api-access-ll2sh\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-kube-api-access-ll2sh\") pod \"02179854-1285-4182-aca7-1e083832067a\" (UID: \"02179854-1285-4182-aca7-1e083832067a\") " Nov 03 20:04:35.245152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243048 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-data" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243541 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-certs" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243726 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-certs" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.243888 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-credential-socket" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.244177 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istiod-ca-cert" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.244415 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-socket" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.244626 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-socket" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.244806 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-data" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.244883 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-envoy" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.244924 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40ba5076-0c89-43ed-a408-0005716f7a87-istio-podinfo" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.244958 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ba5076-0c89-43ed-a408-0005716f7a87-istiod-ca-cert" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:35.245334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.245123 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-credential-socket" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.245925 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.245524 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db981728-088c-4efd-a9b8-6cdee6b42866-istiod-ca-cert" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:35.246290 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246236 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-data" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.246290 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246269 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02179854-1285-4182-aca7-1e083832067a-istiod-ca-cert" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:35.246484 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246408 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-socket\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.246484 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246441 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-socket" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.246484 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246455 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-podinfo\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.246675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246506 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-token\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-token\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.246675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246569 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-envoy\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.246675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246634 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-certs\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.246675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246642 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-socket" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.246675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246663 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-certs\") pod \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\" (UID: \"3c349a0a-69ad-4f14-ba0d-292c4a009b17\") " Nov 03 20:04:35.246922 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246720 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-credential-socket\") pod \"40ba5076-0c89-43ed-a408-0005716f7a87\" (UID: \"40ba5076-0c89-43ed-a408-0005716f7a87\") " Nov 03 20:04:35.246922 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.246750 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-credential-socket\") pod \"db981728-088c-4efd-a9b8-6cdee6b42866\" (UID: \"db981728-088c-4efd-a9b8-6cdee6b42866\") " Nov 03 20:04:35.247326 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247141 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-certs" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.247414 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247338 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-credential-socket" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247896 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247922 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247939 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ba5076-0c89-43ed-a408-0005716f7a87-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247953 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247966 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247980 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.247994 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ba5076-0c89-43ed-a408-0005716f7a87-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248011 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248025 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248040 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248054 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02179854-1285-4182-aca7-1e083832067a-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248066 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248079 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248092 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248104 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.248115 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248116 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.249176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248131 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db981728-088c-4efd-a9b8-6cdee6b42866-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.249176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248144 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02179854-1285-4182-aca7-1e083832067a-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.249176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248156 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.249176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248170 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.249176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248399 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-certs" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.249176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248622 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-envoy" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.249176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.248640 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-credential-socket" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.250018 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.249989 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-podinfo" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:35.250795 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.250774 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-istio-token" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.255044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.254950 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-istio-token" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.255044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.254970 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db981728-088c-4efd-a9b8-6cdee6b42866-istio-podinfo" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:35.255044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.255019 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02179854-1285-4182-aca7-1e083832067a-istio-podinfo" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:35.255044 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.255024 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-kube-api-access-kgw5z" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "kube-api-access-kgw5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.255449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.255419 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-envoy" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.255876 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.255818 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-kube-api-access-ll2sh" pod "02179854-1285-4182-aca7-1e083832067a" (UID: "02179854-1285-4182-aca7-1e083832067a"). InnerVolumeSpecName "kube-api-access-ll2sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.255876 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.255835 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-token" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.256033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.255981 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-istio-token" pod "db981728-088c-4efd-a9b8-6cdee6b42866" (UID: "db981728-088c-4efd-a9b8-6cdee6b42866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.256191 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.256165 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-kube-api-access-5x56b" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "kube-api-access-5x56b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.256913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.256887 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-kube-api-access-5p4f7" pod "40ba5076-0c89-43ed-a408-0005716f7a87" (UID: "40ba5076-0c89-43ed-a408-0005716f7a87"). InnerVolumeSpecName "kube-api-access-5p4f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:35.257979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.257954 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-envoy" pod "3c349a0a-69ad-4f14-ba0d-292c4a009b17" (UID: "3c349a0a-69ad-4f14-ba0d-292c4a009b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:35.289552 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:04:35.289495 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16fca402_db2e_4ed4_8b69_e2a5b97d5d69.slice/crio-29189015f996f4f45766a05aa7b37340405bd96622374df8e1d85e41d7a7336f WatchSource:0}: Error finding container 29189015f996f4f45766a05aa7b37340405bd96622374df8e1d85e41d7a7336f: Status 404 returned error can't find the container with id 29189015f996f4f45766a05aa7b37340405bd96622374df8e1d85e41d7a7336f Nov 03 20:04:35.292247 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.290395 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz"] Nov 03 20:04:35.299733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.296442 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:35.299733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.296532 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:35.299733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.296591 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:04:35.349763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349731 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349762 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349772 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db981728-088c-4efd-a9b8-6cdee6b42866-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349781 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349793 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349801 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349810 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgw5z\" (UniqueName: \"kubernetes.io/projected/db981728-088c-4efd-a9b8-6cdee6b42866-kube-api-access-kgw5z\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349820 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02179854-1285-4182-aca7-1e083832067a-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349829 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349838 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db981728-088c-4efd-a9b8-6cdee6b42866-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349846 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ba5076-0c89-43ed-a408-0005716f7a87-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349854 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349863 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x56b\" (UniqueName: \"kubernetes.io/projected/3c349a0a-69ad-4f14-ba0d-292c4a009b17-kube-api-access-5x56b\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349871 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4f7\" (UniqueName: \"kubernetes.io/projected/40ba5076-0c89-43ed-a408-0005716f7a87-kube-api-access-5p4f7\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349879 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c349a0a-69ad-4f14-ba0d-292c4a009b17-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:35.349907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:35.349887 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll2sh\" (UniqueName: \"kubernetes.io/projected/02179854-1285-4182-aca7-1e083832067a-kube-api-access-ll2sh\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:36.052359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.052298 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs"] Nov 03 20:04:36.063907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.063877 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" event={"ID":"16fca402-db2e-4ed4-8b69-e2a5b97d5d69","Type":"ContainerStarted","Data":"ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894"} Nov 03 20:04:36.065426 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.065397 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" event={"ID":"16fca402-db2e-4ed4-8b69-e2a5b97d5d69","Type":"ContainerStarted","Data":"29189015f996f4f45766a05aa7b37340405bd96622374df8e1d85e41d7a7336f"} Nov 03 20:04:36.065918 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.065875 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" event={"ID":"db981728-088c-4efd-a9b8-6cdee6b42866","Type":"ContainerDied","Data":"528fffe561834249b4bcca32d31063bb3e7f6b75551f8468818789b84ebaf201"} Nov 03 20:04:36.066072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.065928 4314 scope.go:122] "RemoveContainer" containerID="9f3633a327e4aadacde7aea34daa2cb7c66d41a7006445e9397d5720e9b25513" Nov 03 20:04:36.066203 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.065869 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp" Nov 03 20:04:36.080393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.080355 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" event={"ID":"40ba5076-0c89-43ed-a408-0005716f7a87","Type":"ContainerDied","Data":"ebe9c30e23bbc38f416bb16515ce497cf6032ee28e40ebcf950cd2c2f470f38d"} Nov 03 20:04:36.080493 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.080469 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq" Nov 03 20:04:36.087481 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.087466 4314 scope.go:122] "RemoveContainer" containerID="e0871c6f718aa1ed48432c7d8442606c730187cd81269a9e217bf27d2d53f741" Nov 03 20:04:36.089279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.089257 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" event={"ID":"02179854-1285-4182-aca7-1e083832067a","Type":"ContainerDied","Data":"6004dbe48cfb5b1145ebc5c9331bfda3c74128f158408d49ea5c0e7e7a78cf62"} Nov 03 20:04:36.089626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.089607 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9" Nov 03 20:04:36.104565 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.104543 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" event={"ID":"3c349a0a-69ad-4f14-ba0d-292c4a009b17","Type":"ContainerDied","Data":"aa72097a62248796282cebcc178a6a71cf21dad9e07c387ca0d60d36b04e8c0e"} Nov 03 20:04:36.104823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.104767 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp" Nov 03 20:04:36.120591 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.117718 4314 scope.go:122] "RemoveContainer" containerID="1b930b64d1860f96ef932512107a8fa4a9740ba52fb3fcd5896c4d8609741932" Nov 03 20:04:36.121687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.121623 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podStartSLOduration=2.121606263 podStartE2EDuration="2.121606263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:36.117387187 +0000 UTC m=+1983.822226530" watchObservedRunningTime="2026-11-03 20:04:36.121606263 +0000 UTC m=+1983.826445602" Nov 03 20:04:36.206858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.206834 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq"] Nov 03 20:04:36.246628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.246542 4314 scope.go:122] "RemoveContainer" containerID="50fb4ba4b1c76a6cad6f1222e6e9d73862813b0ce39db1cef4e0a1f835aeca23" Nov 03 20:04:36.255003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.247614 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-wsvrq"] Nov 03 20:04:36.255003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.254872 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9"] Nov 03 20:04:36.290864 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.289656 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-rcdz9"] Nov 03 20:04:36.299695 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.299661 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp"] Nov 03 20:04:36.304293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.304228 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-gvqhp"] Nov 03 20:04:36.312020 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.311347 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp"] Nov 03 20:04:36.315539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.315522 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-n9pzp"] Nov 03 20:04:36.429118 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.429087 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02179854-1285-4182-aca7-1e083832067a" path="/var/lib/kubelet/pods/02179854-1285-4182-aca7-1e083832067a/volumes" Nov 03 20:04:36.429657 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.429639 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" path="/var/lib/kubelet/pods/3c349a0a-69ad-4f14-ba0d-292c4a009b17/volumes" Nov 03 20:04:36.430162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.430146 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" path="/var/lib/kubelet/pods/40ba5076-0c89-43ed-a408-0005716f7a87/volumes" Nov 03 20:04:36.430966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.430949 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" path="/var/lib/kubelet/pods/db981728-088c-4efd-a9b8-6cdee6b42866/volumes" Nov 03 20:04:36.507533 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.507504 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77"] Nov 03 20:04:36.508642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508615 4314 cpu_manager.go:414] "containerMap: removing container" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" Nov 03 20:04:36.508642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508635 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" Nov 03 20:04:36.508642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508647 4314 cpu_manager.go:414] "containerMap: removing container" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508654 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508667 4314 cpu_manager.go:414] "containerMap: removing container" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508676 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508692 4314 cpu_manager.go:414] "containerMap: removing container" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508697 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508708 4314 cpu_manager.go:414] "containerMap: removing container" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508713 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508847 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="db981728-088c-4efd-a9b8-6cdee6b42866" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508866 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="02179854-1285-4182-aca7-1e083832067a" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508872 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508879 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c349a0a-69ad-4f14-ba0d-292c4a009b17" containerName="istio-proxy" Nov 03 20:04:36.508966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.508889 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ba5076-0c89-43ed-a408-0005716f7a87" containerName="istio-proxy" Nov 03 20:04:36.515258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.515238 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.516160 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.516141 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77"] Nov 03 20:04:36.519309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.519285 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:36.519422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.519407 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:04:36.598156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.598065 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.598315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.598185 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnct\" (UniqueName: \"kubernetes.io/projected/b57b1a77-c46d-4c91-a5ac-694c6995525b-kube-api-access-sjnct\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.598394 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.598375 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.708624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.707167 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.708624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.707236 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.708624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.707359 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnct\" (UniqueName: \"kubernetes.io/projected/b57b1a77-c46d-4c91-a5ac-694c6995525b-kube-api-access-sjnct\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.708929 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.708697 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.711733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.711707 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.716064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.716037 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnct\" (UniqueName: \"kubernetes.io/projected/b57b1a77-c46d-4c91-a5ac-694c6995525b-kube-api-access-sjnct\") pod \"maas-api-revoke-sub-d87b927e-q9v77\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:36.719419 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.719388 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:36.721461 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.721440 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:04:36.721540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.721484 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:04:36.831573 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:36.831540 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:37.215240 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:37.215202 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77"] Nov 03 20:04:37.220912 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:04:37.220837 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57b1a77_c46d_4c91_a5ac_694c6995525b.slice/crio-d68d3f0260e7ad17c3210bca2f027708479cf96a06dc0340016eb106f4db47c1 WatchSource:0}: Error finding container d68d3f0260e7ad17c3210bca2f027708479cf96a06dc0340016eb106f4db47c1: Status 404 returned error can't find the container with id d68d3f0260e7ad17c3210bca2f027708479cf96a06dc0340016eb106f4db47c1 Nov 03 20:04:37.403456 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:37.403415 4314 scope.go:122] "RemoveContainer" containerID="b70707b1082ef4b21aff627053f4896b50eba458f547b09888b0222db90a69ff" Nov 03 20:04:37.718855 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:37.718827 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:04:37.718986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:37.718882 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:04:38.238755 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:38.238720 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" event={"ID":"b57b1a77-c46d-4c91-a5ac-694c6995525b","Type":"ContainerStarted","Data":"572a94a134fce1f13a408651fa6842092949b9436920ae7bee338894699c7a3b"} Nov 03 20:04:38.238755 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:38.238772 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" event={"ID":"b57b1a77-c46d-4c91-a5ac-694c6995525b","Type":"ContainerStarted","Data":"d68d3f0260e7ad17c3210bca2f027708479cf96a06dc0340016eb106f4db47c1"} Nov 03 20:04:38.275879 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:38.256967 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" podStartSLOduration=2.256949492 podStartE2EDuration="2.256949492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.253779998 +0000 UTC m=+1985.958619339" watchObservedRunningTime="2026-11-03 20:04:38.256949492 +0000 UTC m=+1985.961788835" Nov 03 20:04:38.718737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:38.718705 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:04:38.718892 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:38.718759 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:04:39.238606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.238448 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7"] Nov 03 20:04:39.252601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.243516 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" podUID="0b0afcbf-c491-43d5-a545-08b5b7174777" containerName="maas-api" containerID="cri-o://9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a" gracePeriod=30 Nov 03 20:04:39.252601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.252368 4314 generic.go:356] "Generic (PLEG): container finished" podID="b57b1a77-c46d-4c91-a5ac-694c6995525b" containerID="572a94a134fce1f13a408651fa6842092949b9436920ae7bee338894699c7a3b" exitCode=0 Nov 03 20:04:39.253458 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.253224 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" event={"ID":"b57b1a77-c46d-4c91-a5ac-694c6995525b","Type":"ContainerDied","Data":"572a94a134fce1f13a408651fa6842092949b9436920ae7bee338894699c7a3b"} Nov 03 20:04:39.922629 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.919861 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:04:39.922629 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.919930 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:04:39.932666 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.932622 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l"] Nov 03 20:04:39.945987 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.945960 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d12868-xct7l"] Nov 03 20:04:39.946637 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:39.946612 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 20:04:40.132196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.131195 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-maas-api-tls\") pod \"0b0afcbf-c491-43d5-a545-08b5b7174777\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " Nov 03 20:04:40.132196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.131358 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0afcbf-c491-43d5-a545-08b5b7174777-kube-api-access-wj4b6\" (UniqueName: \"kubernetes.io/projected/0b0afcbf-c491-43d5-a545-08b5b7174777-kube-api-access-wj4b6\") pod \"0b0afcbf-c491-43d5-a545-08b5b7174777\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " Nov 03 20:04:40.132196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.131602 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0afcbf-c491-43d5-a545-08b5b7174777-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0afcbf-c491-43d5-a545-08b5b7174777-openshift-service-ca\") pod \"0b0afcbf-c491-43d5-a545-08b5b7174777\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " Nov 03 20:04:40.132196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.131675 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-metrics-certs\") pod \"0b0afcbf-c491-43d5-a545-08b5b7174777\" (UID: \"0b0afcbf-c491-43d5-a545-08b5b7174777\") " Nov 03 20:04:40.139205 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.139167 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-maas-api-tls" pod "0b0afcbf-c491-43d5-a545-08b5b7174777" (UID: "0b0afcbf-c491-43d5-a545-08b5b7174777"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:40.155935 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.152721 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0afcbf-c491-43d5-a545-08b5b7174777-openshift-service-ca" pod "0b0afcbf-c491-43d5-a545-08b5b7174777" (UID: "0b0afcbf-c491-43d5-a545-08b5b7174777"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:40.166793 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.166744 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0afcbf-c491-43d5-a545-08b5b7174777-kube-api-access-wj4b6" pod "0b0afcbf-c491-43d5-a545-08b5b7174777" (UID: "0b0afcbf-c491-43d5-a545-08b5b7174777"). InnerVolumeSpecName "kube-api-access-wj4b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:40.196605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.191930 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z"] Nov 03 20:04:40.219782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.219727 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-metrics-certs" pod "0b0afcbf-c491-43d5-a545-08b5b7174777" (UID: "0b0afcbf-c491-43d5-a545-08b5b7174777"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:40.234553 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.234522 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b0afcbf-c491-43d5-a545-08b5b7174777-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:40.234773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.234759 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:40.234860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.234849 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b0afcbf-c491-43d5-a545-08b5b7174777-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:40.234952 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.234942 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4b6\" (UniqueName: \"kubernetes.io/projected/0b0afcbf-c491-43d5-a545-08b5b7174777-kube-api-access-wj4b6\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:40.311703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.311665 4314 generic.go:356] "Generic (PLEG): container finished" podID="0b0afcbf-c491-43d5-a545-08b5b7174777" containerID="9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a" exitCode=0 Nov 03 20:04:40.312248 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.312224 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" Nov 03 20:04:40.319271 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.319236 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" event={"ID":"0b0afcbf-c491-43d5-a545-08b5b7174777","Type":"ContainerDied","Data":"9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a"} Nov 03 20:04:40.319391 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.319301 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7" event={"ID":"0b0afcbf-c491-43d5-a545-08b5b7174777","Type":"ContainerDied","Data":"1b486276f47e01001d23bcc6cf2e2214340d7a1c21e73d7a2090f88d8a558820"} Nov 03 20:04:40.319391 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.319335 4314 scope.go:122] "RemoveContainer" containerID="9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a" Nov 03 20:04:40.337792 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.337717 4314 scope.go:122] "RemoveContainer" containerID="9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a" Nov 03 20:04:40.340632 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:40.338171 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a\": container with ID starting with 9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a not found: ID does not exist" containerID="9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a" Nov 03 20:04:40.340632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.338220 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a"} err="failed to get container status \"9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a\": rpc error: code = NotFound desc = could not find container \"9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a\": container with ID starting with 9ae1979f600ce869957ad86ecd4dd0b0507c2e616bc24df59422828a7802f39a not found: ID does not exist" Nov 03 20:04:40.349656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.344060 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7"] Nov 03 20:04:40.351416 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.350746 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d12868-69b7b9b8d4-kxnd7"] Nov 03 20:04:40.436542 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.436516 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0afcbf-c491-43d5-a545-08b5b7174777" path="/var/lib/kubelet/pods/0b0afcbf-c491-43d5-a545-08b5b7174777/volumes" Nov 03 20:04:40.437079 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.437064 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5548e58d-360e-4b10-a177-166cbe7b7924" path="/var/lib/kubelet/pods/5548e58d-360e-4b10-a177-166cbe7b7924/volumes" Nov 03 20:04:40.615701 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.611299 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:40.651626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.644621 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-xks-ca\") pod \"b57b1a77-c46d-4c91-a5ac-694c6995525b\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " Nov 03 20:04:40.651626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.644923 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57b1a77-c46d-4c91-a5ac-694c6995525b-kube-api-access-sjnct\" (UniqueName: \"kubernetes.io/projected/b57b1a77-c46d-4c91-a5ac-694c6995525b-kube-api-access-sjnct\") pod \"b57b1a77-c46d-4c91-a5ac-694c6995525b\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " Nov 03 20:04:40.651626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.644986 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-openshift-ca\") pod \"b57b1a77-c46d-4c91-a5ac-694c6995525b\" (UID: \"b57b1a77-c46d-4c91-a5ac-694c6995525b\") " Nov 03 20:04:40.651626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.647089 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-openshift-ca" pod "b57b1a77-c46d-4c91-a5ac-694c6995525b" (UID: "b57b1a77-c46d-4c91-a5ac-694c6995525b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:40.678394 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.677716 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-xks-ca" pod "b57b1a77-c46d-4c91-a5ac-694c6995525b" (UID: "b57b1a77-c46d-4c91-a5ac-694c6995525b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:40.680141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.680110 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57b1a77-c46d-4c91-a5ac-694c6995525b-kube-api-access-sjnct" pod "b57b1a77-c46d-4c91-a5ac-694c6995525b" (UID: "b57b1a77-c46d-4c91-a5ac-694c6995525b"). InnerVolumeSpecName "kube-api-access-sjnct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:40.730396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.729935 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:04:40.730396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.729999 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:04:40.795905 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.795871 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnct\" (UniqueName: \"kubernetes.io/projected/b57b1a77-c46d-4c91-a5ac-694c6995525b-kube-api-access-sjnct\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:40.809665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.809632 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:40.809888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:40.809867 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57b1a77-c46d-4c91-a5ac-694c6995525b-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:41.414387 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:41.413410 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" event={"ID":"b57b1a77-c46d-4c91-a5ac-694c6995525b","Type":"ContainerDied","Data":"d68d3f0260e7ad17c3210bca2f027708479cf96a06dc0340016eb106f4db47c1"} Nov 03 20:04:41.414387 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:41.413476 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68d3f0260e7ad17c3210bca2f027708479cf96a06dc0340016eb106f4db47c1" Nov 03 20:04:41.414387 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:41.413561 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77" Nov 03 20:04:41.754187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:41.754154 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:04:41.754467 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:41.754436 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:04:41.907674 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:41.907634 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh"] Nov 03 20:04:41.915388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:41.915356 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9f3188-qz2mh"] Nov 03 20:04:42.440359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:42.440324 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de80759-da72-4d5e-bca1-e30fde86e990" path="/var/lib/kubelet/pods/6de80759-da72-4d5e-bca1-e30fde86e990/volumes" Nov 03 20:04:42.718437 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:42.718339 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:04:42.718437 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:42.718398 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:04:42.932264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:42.931746 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" podUID="b92ce758-b31d-46a2-990f-4f222c049a1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:04:43.838357 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:43.838325 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:43.838905 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:43.838880 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:43.839851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:43.839825 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:45.193836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:45.192550 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" podUID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" containerName="payload-processing" containerID="cri-o://3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687" gracePeriod=25 Nov 03 20:04:46.830689 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:46.830651 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx"] Nov 03 20:04:46.831171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:46.830911 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" containerID="cri-o://73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c" gracePeriod=30 Nov 03 20:04:47.515076 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.515042 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q"] Nov 03 20:04:47.515745 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.515729 4314 cpu_manager.go:414] "containerMap: removing container" podUID="0b0afcbf-c491-43d5-a545-08b5b7174777" containerName="maas-api" Nov 03 20:04:47.515816 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.515745 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0afcbf-c491-43d5-a545-08b5b7174777" containerName="maas-api" Nov 03 20:04:47.515816 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.515794 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b57b1a77-c46d-4c91-a5ac-694c6995525b" containerName="revoke-keys" Nov 03 20:04:47.515816 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.515799 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57b1a77-c46d-4c91-a5ac-694c6995525b" containerName="revoke-keys" Nov 03 20:04:47.515951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.515936 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57b1a77-c46d-4c91-a5ac-694c6995525b" containerName="revoke-keys" Nov 03 20:04:47.515995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.515961 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0afcbf-c491-43d5-a545-08b5b7174777" containerName="maas-api" Nov 03 20:04:47.520958 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.520935 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.532684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.532660 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-801162\"/\"auth-test-model-801162-kserve-self-signed-certs\"" Nov 03 20:04:47.532963 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.532948 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-801162\"/\"default-dockercfg-qzps4\"" Nov 03 20:04:47.533528 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.533505 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q"] Nov 03 20:04:47.590602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.589203 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-801162\"/\"kube-root-ca.crt\"" Nov 03 20:04:47.590602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.589356 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-801162\"/\"openshift-service-ca.crt\"" Nov 03 20:04:47.696752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.696712 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-dshm\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.696941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.696765 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tmp-dir\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.697073 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.697052 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzhp\" (UniqueName: \"kubernetes.io/projected/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-kube-api-access-7bzhp\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.697151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.697112 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tls-certs\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.697229 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.697213 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-model-cache\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.697324 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.697308 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-home\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.799682 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.799555 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzhp\" (UniqueName: \"kubernetes.io/projected/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-kube-api-access-7bzhp\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.799682 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.799624 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tls-certs\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.799682 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.799672 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-model-cache\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.799974 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.799866 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-home\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.800049 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.800035 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-dshm\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.800101 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.800072 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tmp-dir\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.800233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.800198 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-model-cache\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.800305 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.800209 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-home\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.800523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.800498 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tmp-dir\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.802457 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.802434 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-dshm\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.802575 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.802459 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tls-certs\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.807714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.807690 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzhp\" (UniqueName: \"kubernetes.io/projected/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-kube-api-access-7bzhp\") pod \"auth-test-model-801162-kserve-fbfb568c7-nd86q\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:47.873605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:47.856898 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:48.324112 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:48.324082 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q"] Nov 03 20:04:48.324438 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:04:48.324416 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48360d9b_b8aa_4a2e_a61e_9287661e9ee8.slice/crio-62c55050f53f7df63f4c1b08e00762cf58c141ee11b5dccc0c7d2d5e498d0f1a WatchSource:0}: Error finding container 62c55050f53f7df63f4c1b08e00762cf58c141ee11b5dccc0c7d2d5e498d0f1a: Status 404 returned error can't find the container with id 62c55050f53f7df63f4c1b08e00762cf58c141ee11b5dccc0c7d2d5e498d0f1a Nov 03 20:04:48.480979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:48.480929 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" event={"ID":"48360d9b-b8aa-4a2e-a61e-9287661e9ee8","Type":"ContainerStarted","Data":"f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c"} Nov 03 20:04:48.481150 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:48.480989 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" event={"ID":"48360d9b-b8aa-4a2e-a61e-9287661e9ee8","Type":"ContainerStarted","Data":"62c55050f53f7df63f4c1b08e00762cf58c141ee11b5dccc0c7d2d5e498d0f1a"} Nov 03 20:04:48.500231 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:48.500178 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" podStartSLOduration=1.500164944 podStartE2EDuration="1.500164944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:48.498652991 +0000 UTC m=+1996.203492331" watchObservedRunningTime="2026-11-03 20:04:48.500164944 +0000 UTC m=+1996.205004283" Nov 03 20:04:48.857493 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:48.857462 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:49.108166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.108094 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 20:04:49.150883 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.150842 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92ce758-b31d-46a2-990f-4f222c049a1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92ce758-b31d-46a2-990f-4f222c049a1f-plugins-config-volume\") pod \"b92ce758-b31d-46a2-990f-4f222c049a1f\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " Nov 03 20:04:49.151107 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.151095 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92ce758-b31d-46a2-990f-4f222c049a1f-kube-api-access-r5j4t\" (UniqueName: \"kubernetes.io/projected/b92ce758-b31d-46a2-990f-4f222c049a1f-kube-api-access-r5j4t\") pod \"b92ce758-b31d-46a2-990f-4f222c049a1f\" (UID: \"b92ce758-b31d-46a2-990f-4f222c049a1f\") " Nov 03 20:04:49.151297 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.151267 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92ce758-b31d-46a2-990f-4f222c049a1f-plugins-config-volume" pod "b92ce758-b31d-46a2-990f-4f222c049a1f" (UID: "b92ce758-b31d-46a2-990f-4f222c049a1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:49.151688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.151651 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92ce758-b31d-46a2-990f-4f222c049a1f-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:49.162601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.153864 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92ce758-b31d-46a2-990f-4f222c049a1f-kube-api-access-r5j4t" pod "b92ce758-b31d-46a2-990f-4f222c049a1f" (UID: "b92ce758-b31d-46a2-990f-4f222c049a1f"). InnerVolumeSpecName "kube-api-access-r5j4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:49.230802 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.230771 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7"] Nov 03 20:04:49.231075 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.231045 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" containerID="cri-o://d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030" gracePeriod=30 Nov 03 20:04:49.238638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.237557 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz"] Nov 03 20:04:49.238638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.237777 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" containerID="cri-o://ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894" gracePeriod=30 Nov 03 20:04:49.252803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.252776 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5j4t\" (UniqueName: \"kubernetes.io/projected/b92ce758-b31d-46a2-990f-4f222c049a1f-kube-api-access-r5j4t\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:49.497391 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.496871 4314 generic.go:356] "Generic (PLEG): container finished" podID="b92ce758-b31d-46a2-990f-4f222c049a1f" containerID="c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0" exitCode=0 Nov 03 20:04:49.498404 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.498181 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" Nov 03 20:04:49.498552 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.498408 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" event={"ID":"b92ce758-b31d-46a2-990f-4f222c049a1f","Type":"ContainerDied","Data":"c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0"} Nov 03 20:04:49.498623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.498549 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx" event={"ID":"b92ce758-b31d-46a2-990f-4f222c049a1f","Type":"ContainerDied","Data":"5f0dc03cb55f7cb168d6cfd5bf094c2ac6a4f51251101169050e78273ee02e75"} Nov 03 20:04:49.500011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.498842 4314 scope.go:122] "RemoveContainer" containerID="c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0" Nov 03 20:04:49.521357 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.521339 4314 scope.go:122] "RemoveContainer" containerID="c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0" Nov 03 20:04:49.521691 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:49.521667 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0\": container with ID starting with c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0 not found: ID does not exist" containerID="c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0" Nov 03 20:04:49.521744 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.521715 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0"} err="failed to get container status \"c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0\": rpc error: code = NotFound desc = could not find container \"c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0\": container with ID starting with c4eeb0adce3f797bb3f831abf29ab1f00ae01dc1a364330b6490d16197260dc0 not found: ID does not exist" Nov 03 20:04:49.531635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.531565 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx"] Nov 03 20:04:49.536510 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.536484 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d12868-598cd4d97f-f6tnx"] Nov 03 20:04:49.710154 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.710119 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8"] Nov 03 20:04:49.711162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.711135 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b92ce758-b31d-46a2-990f-4f222c049a1f" containerName="payload-pre-processing" Nov 03 20:04:49.711162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.711160 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92ce758-b31d-46a2-990f-4f222c049a1f" containerName="payload-pre-processing" Nov 03 20:04:49.711358 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.711345 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92ce758-b31d-46a2-990f-4f222c049a1f" containerName="payload-pre-processing" Nov 03 20:04:49.716653 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.716625 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.721138 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.721095 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8"] Nov 03 20:04:49.724439 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.724399 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:49.728091 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.728068 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:04:49.774501 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.774401 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39075566-7465-46f9-b45d-5172812163a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.774737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.774637 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6gh\" (UniqueName: \"kubernetes.io/projected/39075566-7465-46f9-b45d-5172812163a4-kube-api-access-wg6gh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.774808 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.774777 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39075566-7465-46f9-b45d-5172812163a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.876349 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.876311 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39075566-7465-46f9-b45d-5172812163a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.876525 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.876436 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6gh\" (UniqueName: \"kubernetes.io/projected/39075566-7465-46f9-b45d-5172812163a4-kube-api-access-wg6gh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.876525 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.876520 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39075566-7465-46f9-b45d-5172812163a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.877318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.877289 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39075566-7465-46f9-b45d-5172812163a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.878981 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.878951 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39075566-7465-46f9-b45d-5172812163a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:49.884981 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:49.884953 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6gh\" (UniqueName: \"kubernetes.io/projected/39075566-7465-46f9-b45d-5172812163a4-kube-api-access-wg6gh\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:50.037252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:50.037161 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:50.342202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:50.342171 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8"] Nov 03 20:04:50.342645 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:04:50.342489 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39075566_7465_46f9_b45d_5172812163a4.slice/crio-a059711e598296f26224b71c6e6f54c66af67626516eabb3e9a099d9b7ebb75d WatchSource:0}: Error finding container a059711e598296f26224b71c6e6f54c66af67626516eabb3e9a099d9b7ebb75d: Status 404 returned error can't find the container with id a059711e598296f26224b71c6e6f54c66af67626516eabb3e9a099d9b7ebb75d Nov 03 20:04:50.353334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:50.353308 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:04:50.597197 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:50.597104 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92ce758-b31d-46a2-990f-4f222c049a1f" path="/var/lib/kubelet/pods/b92ce758-b31d-46a2-990f-4f222c049a1f/volumes" Nov 03 20:04:50.604545 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:50.604514 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" event={"ID":"39075566-7465-46f9-b45d-5172812163a4","Type":"ContainerStarted","Data":"a059711e598296f26224b71c6e6f54c66af67626516eabb3e9a099d9b7ebb75d"} Nov 03 20:04:51.093894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.093853 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" containerID="cri-o://4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6" gracePeriod=45 Nov 03 20:04:51.409548 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.409511 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:04:51.553569 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.553539 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-tmp-dir\") pod \"01743f4d-03b7-411a-a125-8224f133fca1\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " Nov 03 20:04:51.553736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.553677 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-dshm\") pod \"01743f4d-03b7-411a-a125-8224f133fca1\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " Nov 03 20:04:51.553840 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.553825 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-home\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-home\") pod \"01743f4d-03b7-411a-a125-8224f133fca1\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " Nov 03 20:04:51.553887 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.553870 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-tmp-dir" pod "01743f4d-03b7-411a-a125-8224f133fca1" (UID: "01743f4d-03b7-411a-a125-8224f133fca1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:51.553975 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.553956 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01743f4d-03b7-411a-a125-8224f133fca1-tls-certs\" (UniqueName: \"kubernetes.io/secret/01743f4d-03b7-411a-a125-8224f133fca1-tls-certs\") pod \"01743f4d-03b7-411a-a125-8224f133fca1\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " Nov 03 20:04:51.554019 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.553999 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-model-cache\") pod \"01743f4d-03b7-411a-a125-8224f133fca1\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " Nov 03 20:04:51.554082 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.554066 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01743f4d-03b7-411a-a125-8224f133fca1-kube-api-access-zfvf7\" (UniqueName: \"kubernetes.io/projected/01743f4d-03b7-411a-a125-8224f133fca1-kube-api-access-zfvf7\") pod \"01743f4d-03b7-411a-a125-8224f133fca1\" (UID: \"01743f4d-03b7-411a-a125-8224f133fca1\") " Nov 03 20:04:51.554130 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.554090 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-home" pod "01743f4d-03b7-411a-a125-8224f133fca1" (UID: "01743f4d-03b7-411a-a125-8224f133fca1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:51.554299 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.554269 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-model-cache" pod "01743f4d-03b7-411a-a125-8224f133fca1" (UID: "01743f4d-03b7-411a-a125-8224f133fca1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:51.554758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.554730 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:51.554758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.554749 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:51.554758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.554759 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:51.555942 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.555919 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-dshm" pod "01743f4d-03b7-411a-a125-8224f133fca1" (UID: "01743f4d-03b7-411a-a125-8224f133fca1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:51.556042 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.556018 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01743f4d-03b7-411a-a125-8224f133fca1-tls-certs" pod "01743f4d-03b7-411a-a125-8224f133fca1" (UID: "01743f4d-03b7-411a-a125-8224f133fca1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:51.556142 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.556120 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01743f4d-03b7-411a-a125-8224f133fca1-kube-api-access-zfvf7" pod "01743f4d-03b7-411a-a125-8224f133fca1" (UID: "01743f4d-03b7-411a-a125-8224f133fca1"). InnerVolumeSpecName "kube-api-access-zfvf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:51.616953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.616873 4314 generic.go:356] "Generic (PLEG): container finished" podID="01743f4d-03b7-411a-a125-8224f133fca1" containerID="4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6" exitCode=0 Nov 03 20:04:51.616953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.616944 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" Nov 03 20:04:51.617110 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.616955 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" event={"ID":"01743f4d-03b7-411a-a125-8224f133fca1","Type":"ContainerDied","Data":"4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6"} Nov 03 20:04:51.617110 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.616999 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs" event={"ID":"01743f4d-03b7-411a-a125-8224f133fca1","Type":"ContainerDied","Data":"f94c58bc5326704285d3c4a7a391ac7bfb1c79f88981f817713cc11b2ac5a3bd"} Nov 03 20:04:51.617110 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.617018 4314 scope.go:122] "RemoveContainer" containerID="4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6" Nov 03 20:04:51.619022 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.618997 4314 generic.go:356] "Generic (PLEG): container finished" podID="39075566-7465-46f9-b45d-5172812163a4" containerID="dc23e2981a709ffe888feefca9c49eae09d17c8f73eec0760aecba36b5fdca1f" exitCode=0 Nov 03 20:04:51.619178 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.619040 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" event={"ID":"39075566-7465-46f9-b45d-5172812163a4","Type":"ContainerDied","Data":"dc23e2981a709ffe888feefca9c49eae09d17c8f73eec0760aecba36b5fdca1f"} Nov 03 20:04:51.629070 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.629046 4314 scope.go:122] "RemoveContainer" containerID="4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6" Nov 03 20:04:51.629380 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:51.629356 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6\": container with ID starting with 4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6 not found: ID does not exist" containerID="4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6" Nov 03 20:04:51.629486 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.629386 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6"} err="failed to get container status \"4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6\": rpc error: code = NotFound desc = could not find container \"4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6\": container with ID starting with 4e54c2af4a6415ec782e8c66b265fb3612a4c2b97d72731c77f64ef82c2020a6 not found: ID does not exist" Nov 03 20:04:51.656270 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.656243 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01743f4d-03b7-411a-a125-8224f133fca1-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:51.656361 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.656276 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvf7\" (UniqueName: \"kubernetes.io/projected/01743f4d-03b7-411a-a125-8224f133fca1-kube-api-access-zfvf7\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:51.656361 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.656292 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01743f4d-03b7-411a-a125-8224f133fca1-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:51.666010 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.665989 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs"] Nov 03 20:04:51.672128 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:51.672102 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3cee6a/test-model-3cee6a-kserve-65d9d4dcf7-h84cs"] Nov 03 20:04:52.141298 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:52.141233 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:04:52.370438 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.370408 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:04:52.454627 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.451879 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01743f4d-03b7-411a-a125-8224f133fca1" path="/var/lib/kubelet/pods/01743f4d-03b7-411a-a125-8224f133fca1/volumes" Nov 03 20:04:52.524159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.524115 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-credential-socket\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.534755 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.534715 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-istio-token\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-istio-token\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.535852 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.535824 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-socket\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.536309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.536289 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-data\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.536493 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.536475 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40337036-b540-4958-af83-99d1c44752c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40337036-b540-4958-af83-99d1c44752c4-istio-podinfo\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.536923 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.536899 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-kube-api-access-mxwwd\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-kube-api-access-mxwwd\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.540827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.540806 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40337036-b540-4958-af83-99d1c44752c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40337036-b540-4958-af83-99d1c44752c4-istiod-ca-cert\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.542445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.536204 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-socket" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:52.542445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.537812 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-credential-socket" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:52.542445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.542055 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40337036-b540-4958-af83-99d1c44752c4-istiod-ca-cert" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:52.542763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.542745 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-envoy\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.543275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.543249 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-certs\") pod \"40337036-b540-4958-af83-99d1c44752c4\" (UID: \"40337036-b540-4958-af83-99d1c44752c4\") " Nov 03 20:04:52.543778 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.543671 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-certs" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:52.543920 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.543903 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-data" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:52.544745 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.544657 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40337036-b540-4958-af83-99d1c44752c4-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.544883 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.544867 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.544975 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.544962 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.545070 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.545059 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.549619 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.549574 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40337036-b540-4958-af83-99d1c44752c4-istio-podinfo" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:52.550170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.550136 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-envoy" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:52.552074 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.552052 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-kube-api-access-mxwwd" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "kube-api-access-mxwwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:52.552905 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.552874 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-istio-token" pod "40337036-b540-4958-af83-99d1c44752c4" (UID: "40337036-b540-4958-af83-99d1c44752c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:52.646312 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.646279 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwwd\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-kube-api-access-mxwwd\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.646312 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.646316 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.646558 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.646330 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40337036-b540-4958-af83-99d1c44752c4-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.646558 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.646345 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40337036-b540-4958-af83-99d1c44752c4-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.646558 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.646358 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40337036-b540-4958-af83-99d1c44752c4-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:52.647454 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.647426 4314 generic.go:356] "Generic (PLEG): container finished" podID="40337036-b540-4958-af83-99d1c44752c4" containerID="73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c" exitCode=0 Nov 03 20:04:52.647575 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.647479 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" Nov 03 20:04:52.647575 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.647506 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" event={"ID":"40337036-b540-4958-af83-99d1c44752c4","Type":"ContainerDied","Data":"73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c"} Nov 03 20:04:52.647575 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.647566 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx" event={"ID":"40337036-b540-4958-af83-99d1c44752c4","Type":"ContainerDied","Data":"f607bd87a97cccb433ffc9cb401aee143eb03cdd436378eac02f49353500fd21"} Nov 03 20:04:52.647768 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.647618 4314 scope.go:122] "RemoveContainer" containerID="73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c" Nov 03 20:04:52.699069 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.699010 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx"] Nov 03 20:04:52.699350 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.699321 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d12868-openshift-default-7f4d86c65f-qrhbx"] Nov 03 20:04:52.715165 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.715135 4314 scope.go:122] "RemoveContainer" containerID="73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c" Nov 03 20:04:52.717724 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:52.717696 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c\": container with ID starting with 73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c not found: ID does not exist" containerID="73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c" Nov 03 20:04:52.717847 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.717736 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c"} err="failed to get container status \"73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c\": rpc error: code = NotFound desc = could not find container \"73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c\": container with ID starting with 73390db232cf49e019b8fc9e60e1451c2d676a9d47d5ff7b0e762b5c54bb889c not found: ID does not exist" Nov 03 20:04:52.845341 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.845314 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:52.952058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.952021 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39075566-7465-46f9-b45d-5172812163a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39075566-7465-46f9-b45d-5172812163a4-maas-api-xks-ca\") pod \"39075566-7465-46f9-b45d-5172812163a4\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " Nov 03 20:04:52.952382 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.952358 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39075566-7465-46f9-b45d-5172812163a4-kube-api-access-wg6gh\" (UniqueName: \"kubernetes.io/projected/39075566-7465-46f9-b45d-5172812163a4-kube-api-access-wg6gh\") pod \"39075566-7465-46f9-b45d-5172812163a4\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " Nov 03 20:04:52.952620 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.952572 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39075566-7465-46f9-b45d-5172812163a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39075566-7465-46f9-b45d-5172812163a4-maas-api-openshift-ca\") pod \"39075566-7465-46f9-b45d-5172812163a4\" (UID: \"39075566-7465-46f9-b45d-5172812163a4\") " Nov 03 20:04:52.954011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.953983 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39075566-7465-46f9-b45d-5172812163a4-maas-api-openshift-ca" pod "39075566-7465-46f9-b45d-5172812163a4" (UID: "39075566-7465-46f9-b45d-5172812163a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:52.959743 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.959718 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39075566-7465-46f9-b45d-5172812163a4-maas-api-xks-ca" pod "39075566-7465-46f9-b45d-5172812163a4" (UID: "39075566-7465-46f9-b45d-5172812163a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:52.960052 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:52.960031 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39075566-7465-46f9-b45d-5172812163a4-kube-api-access-wg6gh" pod "39075566-7465-46f9-b45d-5172812163a4" (UID: "39075566-7465-46f9-b45d-5172812163a4"). InnerVolumeSpecName "kube-api-access-wg6gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:53.054313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:53.054278 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39075566-7465-46f9-b45d-5172812163a4-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:53.054313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:53.054310 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6gh\" (UniqueName: \"kubernetes.io/projected/39075566-7465-46f9-b45d-5172812163a4-kube-api-access-wg6gh\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:53.054479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:53.054330 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39075566-7465-46f9-b45d-5172812163a4-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:53.123929 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:53.123844 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" podUID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:04:53.657905 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:53.657868 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" event={"ID":"39075566-7465-46f9-b45d-5172812163a4","Type":"ContainerDied","Data":"a059711e598296f26224b71c6e6f54c66af67626516eabb3e9a099d9b7ebb75d"} Nov 03 20:04:53.658531 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:53.658509 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a059711e598296f26224b71c6e6f54c66af67626516eabb3e9a099d9b7ebb75d" Nov 03 20:04:53.658531 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:53.657932 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8" Nov 03 20:04:54.445013 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.444976 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40337036-b540-4958-af83-99d1c44752c4" path="/var/lib/kubelet/pods/40337036-b540-4958-af83-99d1c44752c4/volumes" Nov 03 20:04:54.569863 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.569827 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606247 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7076438-f898-42dc-80dc-9af5392701a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7076438-f898-42dc-80dc-9af5392701a5-istio-podinfo\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606357 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-credential-socket\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606417 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-istio-token\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-istio-token\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606465 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-data\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606574 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-socket\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606676 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7076438-f898-42dc-80dc-9af5392701a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7076438-f898-42dc-80dc-9af5392701a5-istiod-ca-cert\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606726 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-certs\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606769 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-envoy\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.608983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.606833 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-kube-api-access-qs2ng\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-kube-api-access-qs2ng\") pod \"a7076438-f898-42dc-80dc-9af5392701a5\" (UID: \"a7076438-f898-42dc-80dc-9af5392701a5\") " Nov 03 20:04:54.610297 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.610040 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-certs" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.610764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.610552 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7076438-f898-42dc-80dc-9af5392701a5-istiod-ca-cert" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:54.611150 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.611095 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-socket" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.612377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.612320 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-credential-socket" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.612631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.612597 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-kube-api-access-qs2ng" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "kube-api-access-qs2ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:54.613420 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.613397 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-data" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.614670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.613646 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-envoy" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.615922 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.615886 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-istio-token" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:54.618762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.617883 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7076438-f898-42dc-80dc-9af5392701a5-istio-podinfo" pod "a7076438-f898-42dc-80dc-9af5392701a5" (UID: "a7076438-f898-42dc-80dc-9af5392701a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:54.623894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.623874 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:54.671564 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.671525 4314 generic.go:356] "Generic (PLEG): container finished" podID="a7076438-f898-42dc-80dc-9af5392701a5" containerID="d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030" exitCode=0 Nov 03 20:04:54.672180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.671552 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" event={"ID":"a7076438-f898-42dc-80dc-9af5392701a5","Type":"ContainerDied","Data":"d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030"} Nov 03 20:04:54.672299 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.672219 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" event={"ID":"a7076438-f898-42dc-80dc-9af5392701a5","Type":"ContainerDied","Data":"0674e9c90c23f30496c59ab1384cea7a369cc0975aac0cce64c21e3481b812e2"} Nov 03 20:04:54.672299 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.672257 4314 scope.go:122] "RemoveContainer" containerID="d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030" Nov 03 20:04:54.672420 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.671692 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7" Nov 03 20:04:54.698453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.697444 4314 generic.go:356] "Generic (PLEG): container finished" podID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerID="ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894" exitCode=0 Nov 03 20:04:54.698453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.698036 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" event={"ID":"16fca402-db2e-4ed4-8b69-e2a5b97d5d69","Type":"ContainerDied","Data":"ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894"} Nov 03 20:04:54.698453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.698125 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" event={"ID":"16fca402-db2e-4ed4-8b69-e2a5b97d5d69","Type":"ContainerDied","Data":"29189015f996f4f45766a05aa7b37340405bd96622374df8e1d85e41d7a7336f"} Nov 03 20:04:54.698453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.698293 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz" Nov 03 20:04:54.712695 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712656 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712701 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712734 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712750 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712766 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7076438-f898-42dc-80dc-9af5392701a5-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712779 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712808 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7076438-f898-42dc-80dc-9af5392701a5-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712822 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs2ng\" (UniqueName: \"kubernetes.io/projected/a7076438-f898-42dc-80dc-9af5392701a5-kube-api-access-qs2ng\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.712860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.712838 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7076438-f898-42dc-80dc-9af5392701a5-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.721008 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.720979 4314 scope.go:122] "RemoveContainer" containerID="5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b" Nov 03 20:04:54.722955 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.722912 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7"] Nov 03 20:04:54.729441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.729414 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-rk8k7"] Nov 03 20:04:54.737723 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.737697 4314 scope.go:122] "RemoveContainer" containerID="d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030" Nov 03 20:04:54.738144 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:54.738114 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030\": container with ID starting with d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030 not found: ID does not exist" containerID="d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030" Nov 03 20:04:54.738215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.738155 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030"} err="failed to get container status \"d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030\": rpc error: code = NotFound desc = could not find container \"d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030\": container with ID starting with d4a73a9718acd0731d74f16a8d07b0fcc3f06837e868281ecfe2ce7af7689030 not found: ID does not exist" Nov 03 20:04:54.738215 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.738180 4314 scope.go:122] "RemoveContainer" containerID="5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b" Nov 03 20:04:54.738472 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:54.738453 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b\": container with ID starting with 5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b not found: ID does not exist" containerID="5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b" Nov 03 20:04:54.738515 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.738478 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b"} err="failed to get container status \"5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b\": rpc error: code = NotFound desc = could not find container \"5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b\": container with ID starting with 5da856ade7f47e3830bcc4e46e490477896f3fd6fbb1c528b89a1406559d1e4b not found: ID does not exist" Nov 03 20:04:54.738515 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.738493 4314 scope.go:122] "RemoveContainer" containerID="ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894" Nov 03 20:04:54.749225 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.749208 4314 scope.go:122] "RemoveContainer" containerID="ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894" Nov 03 20:04:54.749547 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:54.749524 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894\": container with ID starting with ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894 not found: ID does not exist" containerID="ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894" Nov 03 20:04:54.749663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.749555 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894"} err="failed to get container status \"ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894\": rpc error: code = NotFound desc = could not find container \"ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894\": container with ID starting with ccea36968d44537b8f4fe5a017d74b5795438169d833fae955adf39bc93fb894 not found: ID does not exist" Nov 03 20:04:54.782863 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:04:54.782549 4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7076438_f898_42dc_80dc_9af5392701a5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7076438_f898_42dc_80dc_9af5392701a5.slice/crio-0674e9c90c23f30496c59ab1384cea7a369cc0975aac0cce64c21e3481b812e2\": RecentStats: unable to find data in memory cache]" Nov 03 20:04:54.813605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813554 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-envoy\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.813720 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813628 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-certs\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.813720 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813653 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-data\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.813720 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813675 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-token\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-token\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.813923 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813793 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-kube-api-access-5tb8x\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-kube-api-access-5tb8x\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.813923 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813894 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-podinfo\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.814079 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813935 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-certs" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.814079 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813999 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-credential-socket\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.814079 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.814047 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istiod-ca-cert\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.814249 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.813994 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-data" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.814249 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.814174 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-socket\") pod \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\" (UID: \"16fca402-db2e-4ed4-8b69-e2a5b97d5d69\") " Nov 03 20:04:54.814249 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.814197 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-credential-socket" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.814489 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.814466 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-socket" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.814530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.814509 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istiod-ca-cert" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:54.814996 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.814973 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.815057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.815004 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.815057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.815021 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.815057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.815035 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.815057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.815049 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.816157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.816131 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-envoy" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:54.816265 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.816173 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-token" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:54.816435 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.816405 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-kube-api-access-5tb8x" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "kube-api-access-5tb8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:54.816874 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.816859 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-podinfo" pod "16fca402-db2e-4ed4-8b69-e2a5b97d5d69" (UID: "16fca402-db2e-4ed4-8b69-e2a5b97d5d69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:04:54.924608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.919928 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.924608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.919972 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.924608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.919989 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tb8x\" (UniqueName: \"kubernetes.io/projected/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-kube-api-access-5tb8x\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:54.924608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:54.920004 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16fca402-db2e-4ed4-8b69-e2a5b97d5d69-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:55.027752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.027721 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz"] Nov 03 20:04:55.033402 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.033323 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-jc5wz"] Nov 03 20:04:55.430958 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.430877 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm"] Nov 03 20:04:55.431125 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.431081 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" containerID="cri-o://901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3" gracePeriod=30 Nov 03 20:04:55.912473 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.912436 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh"] Nov 03 20:04:55.913170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913153 4314 cpu_manager.go:414] "containerMap: removing container" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" Nov 03 20:04:55.913170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913169 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913181 4314 cpu_manager.go:414] "containerMap: removing container" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913186 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913208 4314 cpu_manager.go:414] "containerMap: removing container" podUID="39075566-7465-46f9-b45d-5172812163a4" containerName="revoke-keys" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913214 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39075566-7465-46f9-b45d-5172812163a4" containerName="revoke-keys" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913226 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913232 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913247 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913252 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913260 4314 cpu_manager.go:414] "containerMap: removing container" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" Nov 03 20:04:55.913275 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913264 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" Nov 03 20:04:55.913597 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913357 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" Nov 03 20:04:55.913597 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913365 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" containerName="istio-proxy" Nov 03 20:04:55.913597 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913376 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="01743f4d-03b7-411a-a125-8224f133fca1" containerName="main" Nov 03 20:04:55.913597 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913384 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="40337036-b540-4958-af83-99d1c44752c4" containerName="istio-proxy" Nov 03 20:04:55.913597 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.913392 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="39075566-7465-46f9-b45d-5172812163a4" containerName="revoke-keys" Nov 03 20:04:55.918889 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.918865 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:55.927510 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.923718 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh"] Nov 03 20:04:55.955481 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.955433 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:55.955666 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.955645 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh59v\" (UniqueName: \"kubernetes.io/projected/543da4e8-f347-4c7b-896f-560bd34018ae-kube-api-access-fh59v\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:55.956486 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.955826 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:55.956737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.956717 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:04:55.960841 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:55.960821 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:04:56.057478 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.057436 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:56.057714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.057608 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:56.057784 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.057747 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh59v\" (UniqueName: \"kubernetes.io/projected/543da4e8-f347-4c7b-896f-560bd34018ae-kube-api-access-fh59v\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:56.058336 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.058299 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:56.060185 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.060160 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:56.066663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.066629 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh59v\" (UniqueName: \"kubernetes.io/projected/543da4e8-f347-4c7b-896f-560bd34018ae-kube-api-access-fh59v\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:56.265238 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.265202 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:56.432877 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.432812 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16fca402-db2e-4ed4-8b69-e2a5b97d5d69" path="/var/lib/kubelet/pods/16fca402-db2e-4ed4-8b69-e2a5b97d5d69/volumes" Nov 03 20:04:56.433941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.433916 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7076438-f898-42dc-80dc-9af5392701a5" path="/var/lib/kubelet/pods/a7076438-f898-42dc-80dc-9af5392701a5/volumes" Nov 03 20:04:56.562299 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.562249 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh"] Nov 03 20:04:56.570235 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:04:56.570201 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543da4e8_f347_4c7b_896f_560bd34018ae.slice/crio-4b4e028269336066e65dd422fbe84c019b64ae9fb2896389587ea6617cfaaaf7 WatchSource:0}: Error finding container 4b4e028269336066e65dd422fbe84c019b64ae9fb2896389587ea6617cfaaaf7: Status 404 returned error can't find the container with id 4b4e028269336066e65dd422fbe84c019b64ae9fb2896389587ea6617cfaaaf7 Nov 03 20:04:56.729068 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:56.729028 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" event={"ID":"543da4e8-f347-4c7b-896f-560bd34018ae","Type":"ContainerStarted","Data":"4b4e028269336066e65dd422fbe84c019b64ae9fb2896389587ea6617cfaaaf7"} Nov 03 20:04:57.739113 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:57.739080 4314 generic.go:356] "Generic (PLEG): container finished" podID="543da4e8-f347-4c7b-896f-560bd34018ae" containerID="9bbed9d386caff2bf561e83dd01e8fd01cdfa13befa6718520a2d2389a9e4ad9" exitCode=0 Nov 03 20:04:57.739565 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:57.739158 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" event={"ID":"543da4e8-f347-4c7b-896f-560bd34018ae","Type":"ContainerDied","Data":"9bbed9d386caff2bf561e83dd01e8fd01cdfa13befa6718520a2d2389a9e4ad9"} Nov 03 20:04:57.867665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:57.860005 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:58.007574 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:58.007451 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:58.761247 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:58.761216 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:04:58.957915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:58.957890 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:04:59.022974 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.022890 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-openshift-ca\") pod \"543da4e8-f347-4c7b-896f-560bd34018ae\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " Nov 03 20:04:59.022974 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.022942 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543da4e8-f347-4c7b-896f-560bd34018ae-kube-api-access-fh59v\" (UniqueName: \"kubernetes.io/projected/543da4e8-f347-4c7b-896f-560bd34018ae-kube-api-access-fh59v\") pod \"543da4e8-f347-4c7b-896f-560bd34018ae\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " Nov 03 20:04:59.023198 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.023073 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-xks-ca\") pod \"543da4e8-f347-4c7b-896f-560bd34018ae\" (UID: \"543da4e8-f347-4c7b-896f-560bd34018ae\") " Nov 03 20:04:59.023319 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.023294 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-openshift-ca" pod "543da4e8-f347-4c7b-896f-560bd34018ae" (UID: "543da4e8-f347-4c7b-896f-560bd34018ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:04:59.023850 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.023821 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.025321 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.025292 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543da4e8-f347-4c7b-896f-560bd34018ae-kube-api-access-fh59v" pod "543da4e8-f347-4c7b-896f-560bd34018ae" (UID: "543da4e8-f347-4c7b-896f-560bd34018ae"). InnerVolumeSpecName "kube-api-access-fh59v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:59.025632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.025563 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-xks-ca" pod "543da4e8-f347-4c7b-896f-560bd34018ae" (UID: "543da4e8-f347-4c7b-896f-560bd34018ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:04:59.125527 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.125489 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh59v\" (UniqueName: \"kubernetes.io/projected/543da4e8-f347-4c7b-896f-560bd34018ae-kube-api-access-fh59v\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.125527 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.125521 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543da4e8-f347-4c7b-896f-560bd34018ae-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:59.759408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.759361 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" event={"ID":"543da4e8-f347-4c7b-896f-560bd34018ae","Type":"ContainerDied","Data":"4b4e028269336066e65dd422fbe84c019b64ae9fb2896389587ea6617cfaaaf7"} Nov 03 20:04:59.759626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.759437 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b4e028269336066e65dd422fbe84c019b64ae9fb2896389587ea6617cfaaaf7" Nov 03 20:04:59.759626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:04:59.759379 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh" Nov 03 20:05:00.761680 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.749661 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 20:05:00.806438 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.806399 4314 generic.go:356] "Generic (PLEG): container finished" podID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerID="901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3" exitCode=0 Nov 03 20:05:00.806701 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.806655 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" event={"ID":"3ac9f8e4-f04d-48af-8f72-d364a9402172","Type":"ContainerDied","Data":"901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3"} Nov 03 20:05:00.806784 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.806736 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" Nov 03 20:05:00.806784 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.806758 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm" event={"ID":"3ac9f8e4-f04d-48af-8f72-d364a9402172","Type":"ContainerDied","Data":"b90d1f8d69ef0d176da27aca38f84fe75318afd4c58ede2934b00f75669fdc4a"} Nov 03 20:05:00.806907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.806795 4314 scope.go:122] "RemoveContainer" containerID="901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3" Nov 03 20:05:00.820751 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.820713 4314 scope.go:122] "RemoveContainer" containerID="2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202" Nov 03 20:05:00.979348 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.978472 4314 scope.go:122] "RemoveContainer" containerID="901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3" Nov 03 20:05:00.979348 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.978624 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-credential-socket\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:00.979348 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.978807 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-envoy\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:00.979348 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.978860 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-data\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:00.979348 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.978898 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-certs\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:00.998606 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:00.994788 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3\": container with ID starting with 901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3 not found: ID does not exist" containerID="901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3" Nov 03 20:05:00.998606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.994840 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3"} err="failed to get container status \"901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3\": rpc error: code = NotFound desc = could not find container \"901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3\": container with ID starting with 901cde5b7621f303a03f76990eca0c9dfc3f0af7f0faa734dbe05291fa3eb6d3 not found: ID does not exist" Nov 03 20:05:00.998606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.994869 4314 scope.go:122] "RemoveContainer" containerID="2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202" Nov 03 20:05:00.998606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.995559 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-credential-socket" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:00.998606 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:00.995636 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202\": container with ID starting with 2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202 not found: ID does not exist" containerID="2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202" Nov 03 20:05:00.998606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:00.995673 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202"} err="failed to get container status \"2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202\": rpc error: code = NotFound desc = could not find container \"2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202\": container with ID starting with 2c55c9697c02851dea014eed6414efc0318c8b94df1cf5cf7f63836545b3c202 not found: ID does not exist" Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.024510 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac9f8e4-f04d-48af-8f72-d364a9402172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac9f8e4-f04d-48af-8f72-d364a9402172-istiod-ca-cert\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.024622 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-podinfo\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.024667 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-kube-api-access-h4h4n\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-kube-api-access-h4h4n\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.024749 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-socket\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.024787 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-token\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-token\") pod \"3ac9f8e4-f04d-48af-8f72-d364a9402172\" (UID: \"3ac9f8e4-f04d-48af-8f72-d364a9402172\") " Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.025642 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.027086 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-certs" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.027338 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-socket" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.027717 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac9f8e4-f04d-48af-8f72-d364a9402172-istiod-ca-cert" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:01.052803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.028072 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-data" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:01.053792 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.053722 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-podinfo" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:01.053930 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.053761 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-token" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:01.054012 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.053786 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-kube-api-access-h4h4n" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "kube-api-access-h4h4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:01.054477 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.054438 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-envoy" pod "3ac9f8e4-f04d-48af-8f72-d364a9402172" (UID: "3ac9f8e4-f04d-48af-8f72-d364a9402172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:01.127252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127221 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.127252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127252 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.127441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127261 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.127441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127269 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.127441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127278 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac9f8e4-f04d-48af-8f72-d364a9402172-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.127441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127287 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac9f8e4-f04d-48af-8f72-d364a9402172-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.127441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127295 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac9f8e4-f04d-48af-8f72-d364a9402172-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.127441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.127304 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4h4n\" (UniqueName: \"kubernetes.io/projected/3ac9f8e4-f04d-48af-8f72-d364a9402172-kube-api-access-h4h4n\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:01.139169 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.138891 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm"] Nov 03 20:05:01.142804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:01.142780 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-hqpxm"] Nov 03 20:05:02.430932 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:02.430894 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" path="/var/lib/kubelet/pods/3ac9f8e4-f04d-48af-8f72-d364a9402172/volumes" Nov 03 20:05:03.123961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:03.123920 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" podUID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:05:05.483057 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.482986 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 20:05:05.607976 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.607945 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76cdce8-912a-419d-adb9-d2b6e8749cd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76cdce8-912a-419d-adb9-d2b6e8749cd3-plugins-config-volume\") pod \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " Nov 03 20:05:05.608181 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.608078 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76cdce8-912a-419d-adb9-d2b6e8749cd3-kube-api-access-8s9tn\" (UniqueName: \"kubernetes.io/projected/a76cdce8-912a-419d-adb9-d2b6e8749cd3-kube-api-access-8s9tn\") pod \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\" (UID: \"a76cdce8-912a-419d-adb9-d2b6e8749cd3\") " Nov 03 20:05:05.608372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.608348 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76cdce8-912a-419d-adb9-d2b6e8749cd3-plugins-config-volume" pod "a76cdce8-912a-419d-adb9-d2b6e8749cd3" (UID: "a76cdce8-912a-419d-adb9-d2b6e8749cd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:05.608640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.608621 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76cdce8-912a-419d-adb9-d2b6e8749cd3-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:05.610347 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.610326 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76cdce8-912a-419d-adb9-d2b6e8749cd3-kube-api-access-8s9tn" pod "a76cdce8-912a-419d-adb9-d2b6e8749cd3" (UID: "a76cdce8-912a-419d-adb9-d2b6e8749cd3"). InnerVolumeSpecName "kube-api-access-8s9tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:05.710431 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.710351 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s9tn\" (UniqueName: \"kubernetes.io/projected/a76cdce8-912a-419d-adb9-d2b6e8749cd3-kube-api-access-8s9tn\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:05.846430 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.846395 4314 generic.go:356] "Generic (PLEG): container finished" podID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" containerID="3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687" exitCode=0 Nov 03 20:05:05.846640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.846445 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" Nov 03 20:05:05.846640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.846473 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" event={"ID":"a76cdce8-912a-419d-adb9-d2b6e8749cd3","Type":"ContainerDied","Data":"3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687"} Nov 03 20:05:05.846640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.846526 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z" event={"ID":"a76cdce8-912a-419d-adb9-d2b6e8749cd3","Type":"ContainerDied","Data":"91e8e77a53d2037d7e8427399fdf2bd3e5f5d1c1c8a7ab3560fbc13b7750e63e"} Nov 03 20:05:05.846640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.846547 4314 scope.go:122] "RemoveContainer" containerID="3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687" Nov 03 20:05:05.857380 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.857361 4314 scope.go:122] "RemoveContainer" containerID="3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687" Nov 03 20:05:05.857669 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:05.857644 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687\": container with ID starting with 3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687 not found: ID does not exist" containerID="3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687" Nov 03 20:05:05.857738 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.857679 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687"} err="failed to get container status \"3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687\": rpc error: code = NotFound desc = could not find container \"3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687\": container with ID starting with 3e2764ee42a13be505970cc89e01f754930e7e1c981bf9ed6d2ae9d284545687 not found: ID does not exist" Nov 03 20:05:05.878207 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.878182 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z"] Nov 03 20:05:05.883000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:05.882970 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f58171-5cc4d546dc-kq97z"] Nov 03 20:05:06.456900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:06.456860 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" path="/var/lib/kubelet/pods/a76cdce8-912a-419d-adb9-d2b6e8749cd3/volumes" Nov 03 20:05:08.486796 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.486756 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8"] Nov 03 20:05:08.490421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.490390 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-32a584-pxwr8"] Nov 03 20:05:08.710981 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.710951 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl"] Nov 03 20:05:08.712176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712147 4314 cpu_manager.go:414] "containerMap: removing container" podUID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" containerName="payload-processing" Nov 03 20:05:08.712176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712175 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" containerName="payload-processing" Nov 03 20:05:08.712376 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712187 4314 cpu_manager.go:414] "containerMap: removing container" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" Nov 03 20:05:08.712376 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712196 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" Nov 03 20:05:08.712376 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712248 4314 cpu_manager.go:414] "containerMap: removing container" podUID="543da4e8-f347-4c7b-896f-560bd34018ae" containerName="revoke-keys" Nov 03 20:05:08.712376 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712256 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543da4e8-f347-4c7b-896f-560bd34018ae" containerName="revoke-keys" Nov 03 20:05:08.712567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712486 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" Nov 03 20:05:08.712567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712509 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76cdce8-912a-419d-adb9-d2b6e8749cd3" containerName="payload-processing" Nov 03 20:05:08.712567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712519 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" Nov 03 20:05:08.712567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712539 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="543da4e8-f347-4c7b-896f-560bd34018ae" containerName="revoke-keys" Nov 03 20:05:08.712567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.712550 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7076438-f898-42dc-80dc-9af5392701a5" containerName="istio-proxy" Nov 03 20:05:08.721651 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.718341 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl"] Nov 03 20:05:08.721651 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.718455 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.731684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.731657 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-b30676\"/\"kube-root-ca.crt\"" Nov 03 20:05:08.731824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.731740 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-b30676\"/\"openshift-service-ca.crt\"" Nov 03 20:05:08.732395 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.732370 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-b30676\"/\"default-dockercfg-m57t9\"" Nov 03 20:05:08.732506 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.732475 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-b30676\"/\"auth-test-model-b30676-kserve-self-signed-certs\"" Nov 03 20:05:08.790175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.790146 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tmp-dir\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.790355 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.790183 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-model-cache\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.790355 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.790326 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tls-certs\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.790479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.790456 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-home\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.790529 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.790494 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5xx\" (UniqueName: \"kubernetes.io/projected/4b19b3c1-36e3-454e-90bc-c8ec8de41799-kube-api-access-2f5xx\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.790661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.790642 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-dshm\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.913602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.903278 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-home\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.913602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.903342 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5xx\" (UniqueName: \"kubernetes.io/projected/4b19b3c1-36e3-454e-90bc-c8ec8de41799-kube-api-access-2f5xx\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.913602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.903415 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-dshm\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.913602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.903618 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tmp-dir\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.913602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.903653 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-model-cache\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.913602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.903753 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tls-certs\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.913602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.904996 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-home\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.921270 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.917910 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tmp-dir\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.946602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.938955 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-model-cache\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.983028 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.982990 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tls-certs\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.983199 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.983001 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5xx\" (UniqueName: \"kubernetes.io/projected/4b19b3c1-36e3-454e-90bc-c8ec8de41799-kube-api-access-2f5xx\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:08.983902 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:08.983876 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-dshm\") pod \"auth-test-model-b30676-kserve-848c9d8b65-hsqzl\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:09.044604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:09.044104 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:09.388157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:09.388128 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl"] Nov 03 20:05:09.992428 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:09.992387 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" event={"ID":"4b19b3c1-36e3-454e-90bc-c8ec8de41799","Type":"ContainerStarted","Data":"b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba"} Nov 03 20:05:09.992824 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:09.992435 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" event={"ID":"4b19b3c1-36e3-454e-90bc-c8ec8de41799","Type":"ContainerStarted","Data":"eea2d16ba62f879caafaac51046d37c0d4a1cb31af43baf339356d41082f3e09"} Nov 03 20:05:10.013398 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:10.013344 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" podStartSLOduration=2.013328881 podStartE2EDuration="2.013328881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:10.009029709 +0000 UTC m=+2017.713869053" watchObservedRunningTime="2026-11-03 20:05:10.013328881 +0000 UTC m=+2017.718168217" Nov 03 20:05:10.045170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:10.045135 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:10.430150 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:10.430070 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39075566-7465-46f9-b45d-5172812163a4" path="/var/lib/kubelet/pods/39075566-7465-46f9-b45d-5172812163a4/volumes" Nov 03 20:05:12.890335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:12.890294 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m"] Nov 03 20:05:12.890898 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:12.890573 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" podUID="6f624939-9997-46de-81ea-11d18972084d" containerName="maas-api" containerID="cri-o://b7615c6a31e805dc029d8b461ba29a818f7c209b19d8eccdf2682b5344d2409f" gracePeriod=30 Nov 03 20:05:13.022373 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.022343 4314 generic.go:356] "Generic (PLEG): container finished" podID="6f624939-9997-46de-81ea-11d18972084d" containerID="b7615c6a31e805dc029d8b461ba29a818f7c209b19d8eccdf2682b5344d2409f" exitCode=0 Nov 03 20:05:13.022505 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.022414 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" event={"ID":"6f624939-9997-46de-81ea-11d18972084d","Type":"ContainerDied","Data":"b7615c6a31e805dc029d8b461ba29a818f7c209b19d8eccdf2682b5344d2409f"} Nov 03 20:05:13.091273 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.090999 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" podUID="6f624939-9997-46de-81ea-11d18972084d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.102:8443/health\": dial tcp 10.134.0.102:8443: connect: connection refused" Nov 03 20:05:13.284682 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.284633 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh"] Nov 03 20:05:13.284934 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.284919 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fa9526-x6mqh"] Nov 03 20:05:13.396156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.396128 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 20:05:13.502043 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.502006 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f624939-9997-46de-81ea-11d18972084d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f624939-9997-46de-81ea-11d18972084d-openshift-service-ca\") pod \"6f624939-9997-46de-81ea-11d18972084d\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " Nov 03 20:05:13.502222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.502087 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f624939-9997-46de-81ea-11d18972084d-kube-api-access-p86tz\" (UniqueName: \"kubernetes.io/projected/6f624939-9997-46de-81ea-11d18972084d-kube-api-access-p86tz\") pod \"6f624939-9997-46de-81ea-11d18972084d\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " Nov 03 20:05:13.502222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.502122 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-metrics-certs\") pod \"6f624939-9997-46de-81ea-11d18972084d\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " Nov 03 20:05:13.502222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.502155 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls\") pod \"6f624939-9997-46de-81ea-11d18972084d\" (UID: \"6f624939-9997-46de-81ea-11d18972084d\") " Nov 03 20:05:13.502446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.502413 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f624939-9997-46de-81ea-11d18972084d-openshift-service-ca" pod "6f624939-9997-46de-81ea-11d18972084d" (UID: "6f624939-9997-46de-81ea-11d18972084d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:13.503362 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.503328 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f624939-9997-46de-81ea-11d18972084d-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:13.504498 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.504466 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f624939-9997-46de-81ea-11d18972084d-kube-api-access-p86tz" pod "6f624939-9997-46de-81ea-11d18972084d" (UID: "6f624939-9997-46de-81ea-11d18972084d"). InnerVolumeSpecName "kube-api-access-p86tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:13.504655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.504514 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls" pod "6f624939-9997-46de-81ea-11d18972084d" (UID: "6f624939-9997-46de-81ea-11d18972084d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:13.504655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.504566 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-metrics-certs" pod "6f624939-9997-46de-81ea-11d18972084d" (UID: "6f624939-9997-46de-81ea-11d18972084d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:13.604858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.604773 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:13.604858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.604806 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f624939-9997-46de-81ea-11d18972084d-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:13.604858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:13.604818 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86tz\" (UniqueName: \"kubernetes.io/projected/6f624939-9997-46de-81ea-11d18972084d-kube-api-access-p86tz\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:14.030794 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:14.030764 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" Nov 03 20:05:14.031281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:14.030796 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m" event={"ID":"6f624939-9997-46de-81ea-11d18972084d","Type":"ContainerDied","Data":"778be5d75b5998cbd1a4f28fc87dade7fb41a59249f09eca8bc4604c78f803a1"} Nov 03 20:05:14.031281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:14.030861 4314 scope.go:122] "RemoveContainer" containerID="b7615c6a31e805dc029d8b461ba29a818f7c209b19d8eccdf2682b5344d2409f" Nov 03 20:05:14.057003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:14.056942 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m"] Nov 03 20:05:14.065668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:14.065644 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fa9526-68d6c6445-9b58m"] Nov 03 20:05:14.432518 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:14.432443 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543da4e8-f347-4c7b-896f-560bd34018ae" path="/var/lib/kubelet/pods/543da4e8-f347-4c7b-896f-560bd34018ae/volumes" Nov 03 20:05:14.433188 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:14.433164 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f624939-9997-46de-81ea-11d18972084d" path="/var/lib/kubelet/pods/6f624939-9997-46de-81ea-11d18972084d/volumes" Nov 03 20:05:15.411334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:15.410234 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm"] Nov 03 20:05:15.411334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:15.410559 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" containerID="cri-o://ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac" gracePeriod=30 Nov 03 20:05:19.044572 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:19.044533 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:19.058461 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:19.058433 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:19.099268 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:19.099239 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:05:20.786425 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.786401 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 20:05:20.850762 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.850723 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp"] Nov 03 20:05:20.857624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.857595 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm"] Nov 03 20:05:20.864703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.864678 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p"] Nov 03 20:05:20.943065 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943027 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-kube-api-access-vrmm7\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-kube-api-access-vrmm7\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943099 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b619441d-0505-47ce-a10c-f24d52df0989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b619441d-0505-47ce-a10c-f24d52df0989-istiod-ca-cert\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943191 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-certs\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943250 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b619441d-0505-47ce-a10c-f24d52df0989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b619441d-0505-47ce-a10c-f24d52df0989-istio-podinfo\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943287 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-envoy\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943388 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-istio-token\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-istio-token\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943531 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943504 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-credential-socket\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943531 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943508 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-certs" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:20.943685 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943540 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b619441d-0505-47ce-a10c-f24d52df0989-istiod-ca-cert" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:20.943685 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943575 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-data\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943685 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943658 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-socket\") pod \"b619441d-0505-47ce-a10c-f24d52df0989\" (UID: \"b619441d-0505-47ce-a10c-f24d52df0989\") " Nov 03 20:05:20.943893 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943872 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-credential-socket" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:20.943992 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.943964 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-data" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:20.944081 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.944055 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-socket" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:20.944844 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.944818 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:20.944962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.944845 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:20.944962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.944860 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b619441d-0505-47ce-a10c-f24d52df0989-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:20.944962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.944873 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:20.944962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.944887 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:20.945823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.945795 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-envoy" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:20.945823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.945805 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b619441d-0505-47ce-a10c-f24d52df0989-istio-podinfo" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:20.945961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.945943 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-istio-token" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:20.946011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:20.945970 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-kube-api-access-vrmm7" pod "b619441d-0505-47ce-a10c-f24d52df0989" (UID: "b619441d-0505-47ce-a10c-f24d52df0989"). InnerVolumeSpecName "kube-api-access-vrmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:21.048187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.046734 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrmm7\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-kube-api-access-vrmm7\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:21.048187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.046768 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b619441d-0505-47ce-a10c-f24d52df0989-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:21.048187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.046782 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b619441d-0505-47ce-a10c-f24d52df0989-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:21.048187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.046795 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b619441d-0505-47ce-a10c-f24d52df0989-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:21.129859 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.129825 4314 generic.go:356] "Generic (PLEG): container finished" podID="b619441d-0505-47ce-a10c-f24d52df0989" containerID="ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac" exitCode=0 Nov 03 20:05:21.130032 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.129877 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" Nov 03 20:05:21.130032 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.129900 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" event={"ID":"b619441d-0505-47ce-a10c-f24d52df0989","Type":"ContainerDied","Data":"ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac"} Nov 03 20:05:21.130032 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.129958 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm" event={"ID":"b619441d-0505-47ce-a10c-f24d52df0989","Type":"ContainerDied","Data":"7b6a10569fd7e47b06e598647a3f1310e91c732a6cb75697af2b407384e0d535"} Nov 03 20:05:21.130032 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.129982 4314 scope.go:122] "RemoveContainer" containerID="ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac" Nov 03 20:05:21.141946 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.141930 4314 scope.go:122] "RemoveContainer" containerID="82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd" Nov 03 20:05:21.152514 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.152492 4314 scope.go:122] "RemoveContainer" containerID="ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac" Nov 03 20:05:21.152846 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:21.152815 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac\": container with ID starting with ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac not found: ID does not exist" containerID="ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac" Nov 03 20:05:21.152955 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.152855 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac"} err="failed to get container status \"ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac\": rpc error: code = NotFound desc = could not find container \"ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac\": container with ID starting with ab26c8120befa4e1d2dfe41086f126fa266355bf40542f23bb7d1e53052312ac not found: ID does not exist" Nov 03 20:05:21.152955 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.152878 4314 scope.go:122] "RemoveContainer" containerID="82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd" Nov 03 20:05:21.153142 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:21.153114 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd\": container with ID starting with 82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd not found: ID does not exist" containerID="82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd" Nov 03 20:05:21.153248 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.153147 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd"} err="failed to get container status \"82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd\": rpc error: code = NotFound desc = could not find container \"82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd\": container with ID starting with 82ee36381ec22d2c629020de5604c5e1483b85a69e945b50771df109cfba17bd not found: ID does not exist" Nov 03 20:05:21.159703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.159665 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm"] Nov 03 20:05:21.163929 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:21.163910 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-32a584-openshift-default-b4b9965cc-cvgzm"] Nov 03 20:05:21.991180 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:21.991139 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:05:22.298174 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.297186 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl"] Nov 03 20:05:22.298174 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.297462 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" containerID="cri-o://8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84" gracePeriod=30 Nov 03 20:05:22.516724 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.512699 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b619441d-0505-47ce-a10c-f24d52df0989" path="/var/lib/kubelet/pods/b619441d-0505-47ce-a10c-f24d52df0989/volumes" Nov 03 20:05:22.825307 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.825271 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv"] Nov 03 20:05:22.825995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.825975 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" Nov 03 20:05:22.825995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.825995 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" Nov 03 20:05:22.826141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.826009 4314 cpu_manager.go:414] "containerMap: removing container" podUID="6f624939-9997-46de-81ea-11d18972084d" containerName="maas-api" Nov 03 20:05:22.826141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.826014 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f624939-9997-46de-81ea-11d18972084d" containerName="maas-api" Nov 03 20:05:22.826141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.826041 4314 cpu_manager.go:414] "containerMap: removing container" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" Nov 03 20:05:22.826141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.826047 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac9f8e4-f04d-48af-8f72-d364a9402172" containerName="istio-proxy" Nov 03 20:05:22.826283 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.826154 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" Nov 03 20:05:22.826283 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.826162 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f624939-9997-46de-81ea-11d18972084d" containerName="maas-api" Nov 03 20:05:22.831258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.831237 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:22.837934 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.837906 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv"] Nov 03 20:05:22.877377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.873064 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:05:22.877377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:22.873122 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:23.001295 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.001264 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlwl\" (UniqueName: \"kubernetes.io/projected/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-kube-api-access-gqlwl\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.001773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.001378 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.001773 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.001627 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.102925 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.102837 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.103100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.102996 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlwl\" (UniqueName: \"kubernetes.io/projected/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-kube-api-access-gqlwl\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.103100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.103075 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.103958 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.103928 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.105431 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.105407 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.111065 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.111042 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlwl\" (UniqueName: \"kubernetes.io/projected/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-kube-api-access-gqlwl\") pod \"maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.151908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.151874 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:23.451138 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:23.451110 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv"] Nov 03 20:05:23.451962 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:05:23.451924 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bf3d7b_bbc2_4f2d_946f_113d908e97d4.slice/crio-7723c5b0d373e4fffe0cb9d29f96e3ff1bf46ef04b6347a299caa5947152ce68 WatchSource:0}: Error finding container 7723c5b0d373e4fffe0cb9d29f96e3ff1bf46ef04b6347a299caa5947152ce68: Status 404 returned error can't find the container with id 7723c5b0d373e4fffe0cb9d29f96e3ff1bf46ef04b6347a299caa5947152ce68 Nov 03 20:05:24.170474 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:24.155741 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" event={"ID":"63bf3d7b-bbc2-4f2d-946f-113d908e97d4","Type":"ContainerStarted","Data":"7b72c334327e5b1573804402fa58ce697e331d8405114613f0faa3301fc0ef9d"} Nov 03 20:05:24.170474 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:24.155808 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" event={"ID":"63bf3d7b-bbc2-4f2d-946f-113d908e97d4","Type":"ContainerStarted","Data":"7723c5b0d373e4fffe0cb9d29f96e3ff1bf46ef04b6347a299caa5947152ce68"} Nov 03 20:05:24.201266 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:24.201198 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" podStartSLOduration=2.201126425 podStartE2EDuration="2.201126425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:24.188936536 +0000 UTC m=+2031.893775877" watchObservedRunningTime="2026-11-03 20:05:24.201126425 +0000 UTC m=+2031.905965766" Nov 03 20:05:24.764537 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:24.764504 4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bf3d7b_bbc2_4f2d_946f_113d908e97d4.slice/crio-conmon-7b72c334327e5b1573804402fa58ce697e331d8405114613f0faa3301fc0ef9d.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:05:25.169216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:25.169175 4314 generic.go:356] "Generic (PLEG): container finished" podID="63bf3d7b-bbc2-4f2d-946f-113d908e97d4" containerID="7b72c334327e5b1573804402fa58ce697e331d8405114613f0faa3301fc0ef9d" exitCode=0 Nov 03 20:05:25.169478 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:25.169327 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" event={"ID":"63bf3d7b-bbc2-4f2d-946f-113d908e97d4","Type":"ContainerDied","Data":"7b72c334327e5b1573804402fa58ce697e331d8405114613f0faa3301fc0ef9d"} Nov 03 20:05:26.308211 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.308175 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn"] Nov 03 20:05:26.318629 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.316727 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z"] Nov 03 20:05:26.428379 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.427996 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:26.465268 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.465196 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-openshift-ca\") pod \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " Nov 03 20:05:26.465862 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.465833 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-kube-api-access-gqlwl\" (UniqueName: \"kubernetes.io/projected/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-kube-api-access-gqlwl\") pod \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " Nov 03 20:05:26.466012 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.465906 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-xks-ca\") pod \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\" (UID: \"63bf3d7b-bbc2-4f2d-946f-113d908e97d4\") " Nov 03 20:05:26.466012 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.465940 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-openshift-ca" pod "63bf3d7b-bbc2-4f2d-946f-113d908e97d4" (UID: "63bf3d7b-bbc2-4f2d-946f-113d908e97d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:26.466569 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.466545 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:26.469623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.468734 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-kube-api-access-gqlwl" pod "63bf3d7b-bbc2-4f2d-946f-113d908e97d4" (UID: "63bf3d7b-bbc2-4f2d-946f-113d908e97d4"). InnerVolumeSpecName "kube-api-access-gqlwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:26.476164 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.476104 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-xks-ca" pod "63bf3d7b-bbc2-4f2d-946f-113d908e97d4" (UID: "63bf3d7b-bbc2-4f2d-946f-113d908e97d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:26.574085 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.574015 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqlwl\" (UniqueName: \"kubernetes.io/projected/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-kube-api-access-gqlwl\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:26.574085 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:26.574056 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63bf3d7b-bbc2-4f2d-946f-113d908e97d4-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.221343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.221310 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" Nov 03 20:05:27.221562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.221344 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv" event={"ID":"63bf3d7b-bbc2-4f2d-946f-113d908e97d4","Type":"ContainerDied","Data":"7723c5b0d373e4fffe0cb9d29f96e3ff1bf46ef04b6347a299caa5947152ce68"} Nov 03 20:05:27.221562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.221417 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7723c5b0d373e4fffe0cb9d29f96e3ff1bf46ef04b6347a299caa5947152ce68" Nov 03 20:05:27.594015 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.593988 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 20:05:27.687050 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687002 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c956c314-2fcc-4302-a6ac-11834c80cf9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c314-2fcc-4302-a6ac-11834c80cf9a-istiod-ca-cert\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687257 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687084 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-kube-api-access-qzzpt\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-kube-api-access-qzzpt\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687257 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687148 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-certs\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687257 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687201 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-socket\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687321 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-envoy\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687354 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-token\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-token\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687434 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-podinfo\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687451 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c956c314-2fcc-4302-a6ac-11834c80cf9a-istiod-ca-cert" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:27.687658 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687461 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-data\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687787 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687694 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-data" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:27.687787 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687627 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-credential-socket\") pod \"c956c314-2fcc-4302-a6ac-11834c80cf9a\" (UID: \"c956c314-2fcc-4302-a6ac-11834c80cf9a\") " Nov 03 20:05:27.687914 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.687788 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-certs" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:27.688453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.688206 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-socket" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:27.688453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.688372 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.688453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.688395 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.688453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.688412 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.688453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.688414 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-credential-socket" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:27.688453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.688428 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c956c314-2fcc-4302-a6ac-11834c80cf9a-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.690318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.690291 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-token" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:27.690946 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.690913 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-envoy" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:27.691423 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.691396 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-kube-api-access-qzzpt" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "kube-api-access-qzzpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:27.691879 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.691852 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-podinfo" pod "c956c314-2fcc-4302-a6ac-11834c80cf9a" (UID: "c956c314-2fcc-4302-a6ac-11834c80cf9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:27.789964 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.789930 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.789964 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.789959 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.789964 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.789969 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c956c314-2fcc-4302-a6ac-11834c80cf9a-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.790251 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.789980 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c956c314-2fcc-4302-a6ac-11834c80cf9a-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:27.790251 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:27.789990 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzzpt\" (UniqueName: \"kubernetes.io/projected/c956c314-2fcc-4302-a6ac-11834c80cf9a-kube-api-access-qzzpt\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:28.230960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.230865 4314 generic.go:356] "Generic (PLEG): container finished" podID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerID="8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84" exitCode=0 Nov 03 20:05:28.230960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.230926 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" Nov 03 20:05:28.231176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.230952 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" event={"ID":"c956c314-2fcc-4302-a6ac-11834c80cf9a","Type":"ContainerDied","Data":"8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84"} Nov 03 20:05:28.231176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.231030 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl" event={"ID":"c956c314-2fcc-4302-a6ac-11834c80cf9a","Type":"ContainerDied","Data":"c94b469a076bcf8d0534ed44416ece5a094571fa9aa6eb83f00f4946afb60d6f"} Nov 03 20:05:28.231176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.231055 4314 scope.go:122] "RemoveContainer" containerID="8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84" Nov 03 20:05:28.243244 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.243225 4314 scope.go:122] "RemoveContainer" containerID="e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47" Nov 03 20:05:28.253993 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.253932 4314 scope.go:122] "RemoveContainer" containerID="8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84" Nov 03 20:05:28.256091 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:28.255731 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84\": container with ID starting with 8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84 not found: ID does not exist" containerID="8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84" Nov 03 20:05:28.256091 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.255821 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84"} err="failed to get container status \"8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84\": rpc error: code = NotFound desc = could not find container \"8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84\": container with ID starting with 8428d4f836f0669b53f3b49cc12eff36c60e693e9f41c3b6ba57bd504886ec84 not found: ID does not exist" Nov 03 20:05:28.256091 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.255847 4314 scope.go:122] "RemoveContainer" containerID="e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47" Nov 03 20:05:28.256091 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.255982 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl"] Nov 03 20:05:28.256313 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:28.256171 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47\": container with ID starting with e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47 not found: ID does not exist" containerID="e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47" Nov 03 20:05:28.256313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.256199 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47"} err="failed to get container status \"e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47\": rpc error: code = NotFound desc = could not find container \"e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47\": container with ID starting with e2e71d0cb222a2fa1dd5dc1943f83fd794ecb508b020a2356e2e74a05c3adc47 not found: ID does not exist" Nov 03 20:05:28.261614 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.261570 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3cee6a-openshift-default-854f54449c-4gzhl"] Nov 03 20:05:28.429340 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:28.429307 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" path="/var/lib/kubelet/pods/c956c314-2fcc-4302-a6ac-11834c80cf9a/volumes" Nov 03 20:05:29.662780 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.662736 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj"] Nov 03 20:05:29.663570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663547 4314 cpu_manager.go:414] "containerMap: removing container" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" Nov 03 20:05:29.663570 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663571 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" Nov 03 20:05:29.663777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663609 4314 cpu_manager.go:414] "containerMap: removing container" podUID="63bf3d7b-bbc2-4f2d-946f-113d908e97d4" containerName="revoke-keys" Nov 03 20:05:29.663777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663616 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bf3d7b-bbc2-4f2d-946f-113d908e97d4" containerName="revoke-keys" Nov 03 20:05:29.663777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663635 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" Nov 03 20:05:29.663777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663643 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" Nov 03 20:05:29.663777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663700 4314 cpu_manager.go:414] "containerMap: removing container" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" Nov 03 20:05:29.663777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663707 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" Nov 03 20:05:29.664084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663883 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" Nov 03 20:05:29.664084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663901 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bf3d7b-bbc2-4f2d-946f-113d908e97d4" containerName="revoke-keys" Nov 03 20:05:29.664084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.663912 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b619441d-0505-47ce-a10c-f24d52df0989" containerName="istio-proxy" Nov 03 20:05:29.674286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.673203 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.686744 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.686715 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj"] Nov 03 20:05:29.719184 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.719107 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snzx\" (UniqueName: \"kubernetes.io/projected/d4007995-7031-440b-ba72-08761b3fd345-kube-api-access-5snzx\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.719333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.719308 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4007995-7031-440b-ba72-08761b3fd345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.719437 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.719357 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4007995-7031-440b-ba72-08761b3fd345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.767681 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.767653 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:29.767681 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.767660 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:05:29.821180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.821145 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4007995-7031-440b-ba72-08761b3fd345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.821388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.821194 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4007995-7031-440b-ba72-08761b3fd345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.821388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.821346 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snzx\" (UniqueName: \"kubernetes.io/projected/d4007995-7031-440b-ba72-08761b3fd345-kube-api-access-5snzx\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.821982 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.821959 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4007995-7031-440b-ba72-08761b3fd345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.823798 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.823777 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4007995-7031-440b-ba72-08761b3fd345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:29.829506 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:29.829486 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snzx\" (UniqueName: \"kubernetes.io/projected/d4007995-7031-440b-ba72-08761b3fd345-kube-api-access-5snzx\") pod \"maas-api-revoke-sub-6ad0f6d1-bxjxj\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:30.016907 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:30.016870 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:30.308430 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:05:30.308395 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4007995_7031_440b_ba72_08761b3fd345.slice/crio-6dd419b9733707ae88fdf94d96769da5cbc8a06ede199761475dd9b8ffe5a836 WatchSource:0}: Error finding container 6dd419b9733707ae88fdf94d96769da5cbc8a06ede199761475dd9b8ffe5a836: Status 404 returned error can't find the container with id 6dd419b9733707ae88fdf94d96769da5cbc8a06ede199761475dd9b8ffe5a836 Nov 03 20:05:30.309718 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:30.309690 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj"] Nov 03 20:05:31.267424 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:31.267079 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" event={"ID":"d4007995-7031-440b-ba72-08761b3fd345","Type":"ContainerStarted","Data":"9af0dd1a141d5d7fe578d89b477bdedb0c53a160b21cfec91a56ed3bdca2bcf1"} Nov 03 20:05:31.267424 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:31.267173 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" event={"ID":"d4007995-7031-440b-ba72-08761b3fd345","Type":"ContainerStarted","Data":"6dd419b9733707ae88fdf94d96769da5cbc8a06ede199761475dd9b8ffe5a836"} Nov 03 20:05:31.287443 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:31.287393 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" podStartSLOduration=2.2873797639999998 podStartE2EDuration="2.287379764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:31.281797325 +0000 UTC m=+2038.986636682" watchObservedRunningTime="2026-11-03 20:05:31.287379764 +0000 UTC m=+2038.992219103" Nov 03 20:05:31.310406 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:31.310372 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" podUID="f838cc11-3e00-41af-9a21-661befef36e3" containerName="payload-processing" containerID="cri-o://211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3" gracePeriod=25 Nov 03 20:05:31.317016 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:31.316983 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" podUID="8e55db36-642a-4799-a476-a15360e67526" containerName="payload-pre-processing" containerID="cri-o://7c03cf39c1419594627414ed248977e9e350fd75f75058083334807c2a5361d2" gracePeriod=25 Nov 03 20:05:32.277156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:32.277120 4314 generic.go:356] "Generic (PLEG): container finished" podID="d4007995-7031-440b-ba72-08761b3fd345" containerID="9af0dd1a141d5d7fe578d89b477bdedb0c53a160b21cfec91a56ed3bdca2bcf1" exitCode=0 Nov 03 20:05:32.277553 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:32.277205 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" event={"ID":"d4007995-7031-440b-ba72-08761b3fd345","Type":"ContainerDied","Data":"9af0dd1a141d5d7fe578d89b477bdedb0c53a160b21cfec91a56ed3bdca2bcf1"} Nov 03 20:05:33.195352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.195309 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn"] Nov 03 20:05:33.200787 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.200754 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs"] Nov 03 20:05:33.666143 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.664767 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:33.746340 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.746302 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4007995-7031-440b-ba72-08761b3fd345-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4007995-7031-440b-ba72-08761b3fd345-maas-api-openshift-ca\") pod \"d4007995-7031-440b-ba72-08761b3fd345\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " Nov 03 20:05:33.746515 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.746352 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4007995-7031-440b-ba72-08761b3fd345-kube-api-access-5snzx\" (UniqueName: \"kubernetes.io/projected/d4007995-7031-440b-ba72-08761b3fd345-kube-api-access-5snzx\") pod \"d4007995-7031-440b-ba72-08761b3fd345\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " Nov 03 20:05:33.746515 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.746502 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4007995-7031-440b-ba72-08761b3fd345-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4007995-7031-440b-ba72-08761b3fd345-maas-api-xks-ca\") pod \"d4007995-7031-440b-ba72-08761b3fd345\" (UID: \"d4007995-7031-440b-ba72-08761b3fd345\") " Nov 03 20:05:33.746737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.746713 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4007995-7031-440b-ba72-08761b3fd345-maas-api-openshift-ca" pod "d4007995-7031-440b-ba72-08761b3fd345" (UID: "d4007995-7031-440b-ba72-08761b3fd345"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:33.747309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.747282 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4007995-7031-440b-ba72-08761b3fd345-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:33.750731 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.750692 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4007995-7031-440b-ba72-08761b3fd345-maas-api-xks-ca" pod "d4007995-7031-440b-ba72-08761b3fd345" (UID: "d4007995-7031-440b-ba72-08761b3fd345"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:33.750853 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.750763 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4007995-7031-440b-ba72-08761b3fd345-kube-api-access-5snzx" pod "d4007995-7031-440b-ba72-08761b3fd345" (UID: "d4007995-7031-440b-ba72-08761b3fd345"). InnerVolumeSpecName "kube-api-access-5snzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:33.848880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.848850 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4007995-7031-440b-ba72-08761b3fd345-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:33.848880 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:33.848876 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5snzx\" (UniqueName: \"kubernetes.io/projected/d4007995-7031-440b-ba72-08761b3fd345-kube-api-access-5snzx\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:34.163663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:34.149832 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6"] Nov 03 20:05:34.163663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:34.150088 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" containerID="cri-o://dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e" gracePeriod=30 Nov 03 20:05:34.440196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:34.440117 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" Nov 03 20:05:34.444173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:34.444137 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj" event={"ID":"d4007995-7031-440b-ba72-08761b3fd345","Type":"ContainerDied","Data":"6dd419b9733707ae88fdf94d96769da5cbc8a06ede199761475dd9b8ffe5a836"} Nov 03 20:05:34.444337 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:34.444187 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd419b9733707ae88fdf94d96769da5cbc8a06ede199761475dd9b8ffe5a836" Nov 03 20:05:35.888103 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:35.888017 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" containerID="cri-o://0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71" gracePeriod=45 Nov 03 20:05:35.892747 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:35.892716 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" containerID="cri-o://548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2" gracePeriod=45 Nov 03 20:05:35.896772 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:35.896746 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" containerID="cri-o://d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7" gracePeriod=45 Nov 03 20:05:36.313602 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.313020 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 20:05:36.327274 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.327247 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:05:36.340709 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.340682 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354219 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06bf6477-c859-4d11-8de8-aff72c7d849d-tls-certs\" (UniqueName: \"kubernetes.io/secret/06bf6477-c859-4d11-8de8-aff72c7d849d-tls-certs\") pod \"06bf6477-c859-4d11-8de8-aff72c7d849d\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354265 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-model-cache\") pod \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354305 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad274c2-4168-4b70-a353-52475635630d-tls-certs\" (UniqueName: \"kubernetes.io/secret/aad274c2-4168-4b70-a353-52475635630d-tls-certs\") pod \"aad274c2-4168-4b70-a353-52475635630d\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354354 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad274c2-4168-4b70-a353-52475635630d-kube-api-access-vn6c8\" (UniqueName: \"kubernetes.io/projected/aad274c2-4168-4b70-a353-52475635630d-kube-api-access-vn6c8\") pod \"aad274c2-4168-4b70-a353-52475635630d\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354418 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-tmp-dir\") pod \"06bf6477-c859-4d11-8de8-aff72c7d849d\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354460 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-dshm\") pod \"06bf6477-c859-4d11-8de8-aff72c7d849d\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354486 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bf6477-c859-4d11-8de8-aff72c7d849d-kube-api-access-9hdjm\" (UniqueName: \"kubernetes.io/projected/06bf6477-c859-4d11-8de8-aff72c7d849d-kube-api-access-9hdjm\") pod \"06bf6477-c859-4d11-8de8-aff72c7d849d\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354516 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-home\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-home\") pod \"06bf6477-c859-4d11-8de8-aff72c7d849d\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354544 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-model-cache\") pod \"06bf6477-c859-4d11-8de8-aff72c7d849d\" (UID: \"06bf6477-c859-4d11-8de8-aff72c7d849d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354610 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tmp-dir\") pod \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354688 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-kube-api-access-4hjgc\" (UniqueName: \"kubernetes.io/projected/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-kube-api-access-4hjgc\") pod \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354780 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-home\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-home\") pod \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.354813 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-model-cache\") pod \"aad274c2-4168-4b70-a353-52475635630d\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.355380 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tls-certs\") pod \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.355437 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-tmp-dir\") pod \"aad274c2-4168-4b70-a353-52475635630d\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.355508 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-dshm\") pod \"aad274c2-4168-4b70-a353-52475635630d\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.355539 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-dshm\") pod \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\" (UID: \"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d\") " Nov 03 20:05:36.356408 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.355619 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-home\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-home\") pod \"aad274c2-4168-4b70-a353-52475635630d\" (UID: \"aad274c2-4168-4b70-a353-52475635630d\") " Nov 03 20:05:36.406644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.405575 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bf6477-c859-4d11-8de8-aff72c7d849d-kube-api-access-9hdjm" pod "06bf6477-c859-4d11-8de8-aff72c7d849d" (UID: "06bf6477-c859-4d11-8de8-aff72c7d849d"). InnerVolumeSpecName "kube-api-access-9hdjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:36.406644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.405914 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-home" pod "06bf6477-c859-4d11-8de8-aff72c7d849d" (UID: "06bf6477-c859-4d11-8de8-aff72c7d849d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.406644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.406113 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-model-cache" pod "06bf6477-c859-4d11-8de8-aff72c7d849d" (UID: "06bf6477-c859-4d11-8de8-aff72c7d849d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.406644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.406301 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tmp-dir" pod "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" (UID: "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.406975 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.406729 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-dshm" pod "06bf6477-c859-4d11-8de8-aff72c7d849d" (UID: "06bf6477-c859-4d11-8de8-aff72c7d849d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.407974 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.407942 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-home" pod "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" (UID: "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.408165 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.408140 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-model-cache" pod "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" (UID: "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.408359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.408335 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-tmp-dir" pod "aad274c2-4168-4b70-a353-52475635630d" (UID: "aad274c2-4168-4b70-a353-52475635630d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.408555 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.408533 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-model-cache" pod "aad274c2-4168-4b70-a353-52475635630d" (UID: "aad274c2-4168-4b70-a353-52475635630d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.409781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.409677 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-tmp-dir" pod "06bf6477-c859-4d11-8de8-aff72c7d849d" (UID: "06bf6477-c859-4d11-8de8-aff72c7d849d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.409781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.409737 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-home" pod "aad274c2-4168-4b70-a353-52475635630d" (UID: "aad274c2-4168-4b70-a353-52475635630d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.411441 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.411358 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-kube-api-access-4hjgc" pod "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" (UID: "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d"). InnerVolumeSpecName "kube-api-access-4hjgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:36.411683 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.411652 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tls-certs" pod "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" (UID: "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:36.411763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.411713 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-dshm" pod "aad274c2-4168-4b70-a353-52475635630d" (UID: "aad274c2-4168-4b70-a353-52475635630d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.413656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.413630 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bf6477-c859-4d11-8de8-aff72c7d849d-tls-certs" pod "06bf6477-c859-4d11-8de8-aff72c7d849d" (UID: "06bf6477-c859-4d11-8de8-aff72c7d849d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:36.414795 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.414700 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad274c2-4168-4b70-a353-52475635630d-kube-api-access-vn6c8" pod "aad274c2-4168-4b70-a353-52475635630d" (UID: "aad274c2-4168-4b70-a353-52475635630d"). InnerVolumeSpecName "kube-api-access-vn6c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:36.414795 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.414772 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-dshm" pod "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" (UID: "4bb057b6-7660-4a8e-9cd5-3ffe086cd72d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:36.415300 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.415265 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad274c2-4168-4b70-a353-52475635630d-tls-certs" pod "aad274c2-4168-4b70-a353-52475635630d" (UID: "aad274c2-4168-4b70-a353-52475635630d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467018 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467055 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467070 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdjm\" (UniqueName: \"kubernetes.io/projected/06bf6477-c859-4d11-8de8-aff72c7d849d-kube-api-access-9hdjm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467086 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467101 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bf6477-c859-4d11-8de8-aff72c7d849d-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467114 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467128 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjgc\" (UniqueName: \"kubernetes.io/projected/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-kube-api-access-4hjgc\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467142 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467156 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467168 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467181 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467196 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467208 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467220 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad274c2-4168-4b70-a353-52475635630d-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467232 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bf6477-c859-4d11-8de8-aff72c7d849d-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467245 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.472643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.467260 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad274c2-4168-4b70-a353-52475635630d-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.498874 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.498830 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn6c8\" (UniqueName: \"kubernetes.io/projected/aad274c2-4168-4b70-a353-52475635630d-kube-api-access-vn6c8\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:36.514007 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.513967 4314 generic.go:356] "Generic (PLEG): container finished" podID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerID="548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2" exitCode=0 Nov 03 20:05:36.514200 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.514125 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" event={"ID":"06bf6477-c859-4d11-8de8-aff72c7d849d","Type":"ContainerDied","Data":"548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2"} Nov 03 20:05:36.514200 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.514162 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" event={"ID":"06bf6477-c859-4d11-8de8-aff72c7d849d","Type":"ContainerDied","Data":"aaa43838153bd914aca6d73396e1084cd697399deb8062eb89f1f56688af5a1e"} Nov 03 20:05:36.514200 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.514182 4314 scope.go:122] "RemoveContainer" containerID="548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2" Nov 03 20:05:36.514651 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.514366 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" Nov 03 20:05:36.517345 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.517319 4314 generic.go:356] "Generic (PLEG): container finished" podID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerID="d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7" exitCode=0 Nov 03 20:05:36.517457 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.517407 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" event={"ID":"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d","Type":"ContainerDied","Data":"d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7"} Nov 03 20:05:36.517457 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.517437 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" event={"ID":"4bb057b6-7660-4a8e-9cd5-3ffe086cd72d","Type":"ContainerDied","Data":"4a30e7c6d8868de704e6666aa719039703d4b52967cb208596f5fd02022ad2cd"} Nov 03 20:05:36.517625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.517493 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p" Nov 03 20:05:36.521684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.521663 4314 generic.go:356] "Generic (PLEG): container finished" podID="aad274c2-4168-4b70-a353-52475635630d" containerID="0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71" exitCode=0 Nov 03 20:05:36.521805 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.521713 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" Nov 03 20:05:36.521805 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.521782 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" event={"ID":"aad274c2-4168-4b70-a353-52475635630d","Type":"ContainerDied","Data":"0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71"} Nov 03 20:05:36.521922 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.521814 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" event={"ID":"aad274c2-4168-4b70-a353-52475635630d","Type":"ContainerDied","Data":"a7e3308848f2953809416e51fe5c5fe4b5e8c9b239e915e9ee131481f0e7b8de"} Nov 03 20:05:36.530667 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.530650 4314 scope.go:122] "RemoveContainer" containerID="548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2" Nov 03 20:05:36.530929 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:36.530902 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2\": container with ID starting with 548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2 not found: ID does not exist" containerID="548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2" Nov 03 20:05:36.531019 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.530939 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2"} err="failed to get container status \"548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2\": rpc error: code = NotFound desc = could not find container \"548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2\": container with ID starting with 548d2d09b706ad82869e727525f96042567620432c62d8e1e12616e82f907fa2 not found: ID does not exist" Nov 03 20:05:36.531019 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.530963 4314 scope.go:122] "RemoveContainer" containerID="d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7" Nov 03 20:05:36.540758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.540724 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm"] Nov 03 20:05:36.544798 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.544778 4314 scope.go:122] "RemoveContainer" containerID="d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7" Nov 03 20:05:36.545057 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:36.545037 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7\": container with ID starting with d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7 not found: ID does not exist" containerID="d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7" Nov 03 20:05:36.545135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.545064 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7"} err="failed to get container status \"d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7\": rpc error: code = NotFound desc = could not find container \"d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7\": container with ID starting with d6739ffdab9e62f9fe2dc217d26e98f43716bcad2da8fcbbbbe67e3e456f7bb7 not found: ID does not exist" Nov 03 20:05:36.545135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.545080 4314 scope.go:122] "RemoveContainer" containerID="0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71" Nov 03 20:05:36.546189 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.546171 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm"] Nov 03 20:05:36.554428 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.554410 4314 scope.go:122] "RemoveContainer" containerID="0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71" Nov 03 20:05:36.554692 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:36.554670 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71\": container with ID starting with 0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71 not found: ID does not exist" containerID="0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71" Nov 03 20:05:36.554786 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.554697 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71"} err="failed to get container status \"0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71\": rpc error: code = NotFound desc = could not find container \"0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71\": container with ID starting with 0a05a662dd882c01afbaba85dd358141c951a71e2da22e9a8a516e5008e87d71 not found: ID does not exist" Nov 03 20:05:36.565606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.560206 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p"] Nov 03 20:05:36.592231 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.592207 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/premium-w3-39043d-kserve-846bcd8466-sls7p"] Nov 03 20:05:36.614533 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.614512 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp"] Nov 03 20:05:36.617806 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:36.617786 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp"] Nov 03 20:05:37.151798 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:37.151745 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-w3-39043d-kserve-54b776bfcc-llzcm" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:05:37.152290 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:37.152074 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39043d/distinct-2-w3-39043d-kserve-75c6595994-frbsp" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Nov 03 20:05:38.196085 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.196041 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" podUID="e70b7ba4-f07c-4b53-89b9-80bb29186ad3" containerName="payload-processing" containerID="cri-o://9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711" gracePeriod=25 Nov 03 20:05:38.201401 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.201362 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerName="payload-pre-processing" containerID="cri-o://e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657" gracePeriod=25 Nov 03 20:05:38.437043 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.437009 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" path="/var/lib/kubelet/pods/06bf6477-c859-4d11-8de8-aff72c7d849d/volumes" Nov 03 20:05:38.437976 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.437952 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" path="/var/lib/kubelet/pods/4bb057b6-7660-4a8e-9cd5-3ffe086cd72d/volumes" Nov 03 20:05:38.438728 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.438708 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad274c2-4168-4b70-a353-52475635630d" path="/var/lib/kubelet/pods/aad274c2-4168-4b70-a353-52475635630d/volumes" Nov 03 20:05:38.468503 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.468046 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:05:38.529060 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.529030 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 20:05:38.542037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.542009 4314 generic.go:356] "Generic (PLEG): container finished" podID="e70b7ba4-f07c-4b53-89b9-80bb29186ad3" containerID="9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711" exitCode=0 Nov 03 20:05:38.542151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.542051 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" Nov 03 20:05:38.542151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.542066 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" event={"ID":"e70b7ba4-f07c-4b53-89b9-80bb29186ad3","Type":"ContainerDied","Data":"9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711"} Nov 03 20:05:38.542151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.542104 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" event={"ID":"e70b7ba4-f07c-4b53-89b9-80bb29186ad3","Type":"ContainerDied","Data":"4f2cdea11fdfe2a0c87265481b03a7c57b20f18d72335863a1211254b99cf570"} Nov 03 20:05:38.542151 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.542126 4314 scope.go:122] "RemoveContainer" containerID="9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711" Nov 03 20:05:38.558265 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.557840 4314 scope.go:122] "RemoveContainer" containerID="9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711" Nov 03 20:05:38.558265 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:38.558163 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711\": container with ID starting with 9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711 not found: ID does not exist" containerID="9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711" Nov 03 20:05:38.558265 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.558195 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711"} err="failed to get container status \"9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711\": rpc error: code = NotFound desc = could not find container \"9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711\": container with ID starting with 9437f59381ba7bf86f352bdebca97a1845964dc6c2fdb70447858cc9224b8711 not found: ID does not exist" Nov 03 20:05:38.633138 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.633098 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-plugins-config-volume\") pod \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " Nov 03 20:05:38.633313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.633192 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-kube-api-access-b7z4w\" (UniqueName: \"kubernetes.io/projected/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-kube-api-access-b7z4w\") pod \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\" (UID: \"e70b7ba4-f07c-4b53-89b9-80bb29186ad3\") " Nov 03 20:05:38.633576 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.633549 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-plugins-config-volume" pod "e70b7ba4-f07c-4b53-89b9-80bb29186ad3" (UID: "e70b7ba4-f07c-4b53-89b9-80bb29186ad3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:38.633986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.633961 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:38.635354 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.635330 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-kube-api-access-b7z4w" pod "e70b7ba4-f07c-4b53-89b9-80bb29186ad3" (UID: "e70b7ba4-f07c-4b53-89b9-80bb29186ad3"). InnerVolumeSpecName "kube-api-access-b7z4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:38.735944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.735910 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7z4w\" (UniqueName: \"kubernetes.io/projected/e70b7ba4-f07c-4b53-89b9-80bb29186ad3-kube-api-access-b7z4w\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:38.872437 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.872401 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn"] Nov 03 20:05:38.876604 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:38.876560 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn"] Nov 03 20:05:39.163392 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.163340 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" podUID="8e55db36-642a-4799-a476-a15360e67526" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:05:39.163934 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.163906 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" podUID="f838cc11-3e00-41af-9a21-661befef36e3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:05:39.467300 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.467203 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c18bea-7957fddcf9-qxzkn" podUID="e70b7ba4-f07c-4b53-89b9-80bb29186ad3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.125:50052\" within 1s: context deadline exceeded" Nov 03 20:05:39.467788 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:05:39.467653 4314 logging.go:55] [core] [Channel #3896 SubChannel #3898]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.125:50052", ServerName: "10.134.0.125:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.125:50052: operation was canceled" Nov 03 20:05:39.522312 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.522291 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 20:05:39.557407 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.557375 4314 generic.go:356] "Generic (PLEG): container finished" podID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerID="dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e" exitCode=0 Nov 03 20:05:39.557622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.557427 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" Nov 03 20:05:39.557622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.557459 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" event={"ID":"064de3ea-3b4f-483d-8468-62da6e0a19b7","Type":"ContainerDied","Data":"dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e"} Nov 03 20:05:39.557622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.557521 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6" event={"ID":"064de3ea-3b4f-483d-8468-62da6e0a19b7","Type":"ContainerDied","Data":"79dae26d840939817a203a3ddb0f98f895f4201007f9bb4e3bc9e54a0b0e213f"} Nov 03 20:05:39.557622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.557555 4314 scope.go:122] "RemoveContainer" containerID="dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e" Nov 03 20:05:39.571052 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.571028 4314 scope.go:122] "RemoveContainer" containerID="e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce" Nov 03 20:05:39.587939 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.587693 4314 scope.go:122] "RemoveContainer" containerID="dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e" Nov 03 20:05:39.588439 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:39.588191 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e\": container with ID starting with dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e not found: ID does not exist" containerID="dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e" Nov 03 20:05:39.588439 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.588227 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e"} err="failed to get container status \"dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e\": rpc error: code = NotFound desc = could not find container \"dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e\": container with ID starting with dca8929b0804a9faff8fd428b767e069c68aaf7a70f65ed07311f94ac25f468e not found: ID does not exist" Nov 03 20:05:39.588439 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.588252 4314 scope.go:122] "RemoveContainer" containerID="e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce" Nov 03 20:05:39.590544 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:39.590449 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce\": container with ID starting with e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce not found: ID does not exist" containerID="e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce" Nov 03 20:05:39.590544 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.590484 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce"} err="failed to get container status \"e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce\": rpc error: code = NotFound desc = could not find container \"e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce\": container with ID starting with e8d9c9455c0be47f7616d18541a2704e4e22368b8d4f2a8086d3e240e21686ce not found: ID does not exist" Nov 03 20:05:39.591283 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.591249 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff"] Nov 03 20:05:39.614968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.614941 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4"] Nov 03 20:05:39.648374 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648349 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-credential-socket\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.648538 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648448 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-token\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-token\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.648538 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648485 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-envoy\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.648538 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648531 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064de3ea-3b4f-483d-8468-62da6e0a19b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064de3ea-3b4f-483d-8468-62da6e0a19b7-istiod-ca-cert\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.648814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648551 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-certs\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.648814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648614 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-podinfo\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.648814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648671 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-data\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.648814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648673 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-credential-socket" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:39.648814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648753 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-socket\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.649104 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648819 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-kube-api-access-sx4lw\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-kube-api-access-sx4lw\") pod \"064de3ea-3b4f-483d-8468-62da6e0a19b7\" (UID: \"064de3ea-3b4f-483d-8468-62da6e0a19b7\") " Nov 03 20:05:39.649104 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.648961 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-certs" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:39.649104 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.649018 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-data" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:39.649328 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.649301 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-socket" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:39.649512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.649490 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064de3ea-3b4f-483d-8468-62da6e0a19b7-istiod-ca-cert" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:39.649610 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.649512 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.649610 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.649533 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.649610 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.649548 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.649610 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.649557 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.651211 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.651189 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-token" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:39.651211 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.651198 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-envoy" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:39.651322 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.651211 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-podinfo" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:39.651879 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.651860 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-kube-api-access-sx4lw" pod "064de3ea-3b4f-483d-8468-62da6e0a19b7" (UID: "064de3ea-3b4f-483d-8468-62da6e0a19b7"). InnerVolumeSpecName "kube-api-access-sx4lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:39.751468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.751433 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx4lw\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-kube-api-access-sx4lw\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.751655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.751469 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.751655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.751487 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.751655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.751502 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/064de3ea-3b4f-483d-8468-62da6e0a19b7-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.751655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.751517 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/064de3ea-3b4f-483d-8468-62da6e0a19b7-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:39.895985 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.895902 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6"] Nov 03 20:05:39.899961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:39.899935 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-2d78ea-openshift-default-694d949bdf-xt9q6"] Nov 03 20:05:40.429117 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:40.429083 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" path="/var/lib/kubelet/pods/064de3ea-3b4f-483d-8468-62da6e0a19b7/volumes" Nov 03 20:05:40.429597 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:40.429569 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70b7ba4-f07c-4b53-89b9-80bb29186ad3" path="/var/lib/kubelet/pods/e70b7ba4-f07c-4b53-89b9-80bb29186ad3/volumes" Nov 03 20:05:43.606887 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:43.606851 4314 generic.go:356] "Generic (PLEG): container finished" podID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerID="24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937" exitCode=0 Nov 03 20:05:43.607446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:43.606926 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" event={"ID":"6612b77c-c2f2-4eb4-a203-2ab47761d23b","Type":"ContainerDied","Data":"24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937"} Nov 03 20:05:43.607446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:43.607428 4314 scope.go:122] "RemoveContainer" containerID="24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937" Nov 03 20:05:43.609230 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:43.609197 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:05:43.609345 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:43.609272 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:05:43.609345 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:43.609336 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:05:44.037065 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:44.037033 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv"] Nov 03 20:05:44.040540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:44.040507 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c18bea-vvksv"] Nov 03 20:05:44.428865 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:44.428773 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bf3d7b-bbc2-4f2d-946f-113d908e97d4" path="/var/lib/kubelet/pods/63bf3d7b-bbc2-4f2d-946f-113d908e97d4/volumes" Nov 03 20:05:44.635829 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:44.635792 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" event={"ID":"6612b77c-c2f2-4eb4-a203-2ab47761d23b","Type":"ContainerStarted","Data":"9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2"} Nov 03 20:05:44.934895 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:44.934860 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq"] Nov 03 20:05:45.017900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:45.017851 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:05:45.022301 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:45.022254 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:05:45.022454 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:45.022330 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:05:45.746048 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:45.746008 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7"] Nov 03 20:05:45.746765 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:45.746271 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" podUID="b49683a7-346c-4ce5-8710-699b657febff" containerName="maas-api" containerID="cri-o://c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657" gracePeriod=30 Nov 03 20:05:46.014953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.014778 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:05:46.015256 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.015138 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:05:46.286657 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.276960 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:05:46.414192 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.412107 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 20:05:46.516099 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.516056 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49683a7-346c-4ce5-8710-699b657febff-kube-api-access-xqc4p\" (UniqueName: \"kubernetes.io/projected/b49683a7-346c-4ce5-8710-699b657febff-kube-api-access-xqc4p\") pod \"b49683a7-346c-4ce5-8710-699b657febff\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " Nov 03 20:05:46.516529 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.516508 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-metrics-certs\") pod \"b49683a7-346c-4ce5-8710-699b657febff\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " Nov 03 20:05:46.516881 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.516858 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49683a7-346c-4ce5-8710-699b657febff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49683a7-346c-4ce5-8710-699b657febff-openshift-service-ca\") pod \"b49683a7-346c-4ce5-8710-699b657febff\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " Nov 03 20:05:46.516989 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.516936 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-maas-api-tls\") pod \"b49683a7-346c-4ce5-8710-699b657febff\" (UID: \"b49683a7-346c-4ce5-8710-699b657febff\") " Nov 03 20:05:46.517353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.517321 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49683a7-346c-4ce5-8710-699b657febff-openshift-service-ca" pod "b49683a7-346c-4ce5-8710-699b657febff" (UID: "b49683a7-346c-4ce5-8710-699b657febff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:46.517615 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.517569 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49683a7-346c-4ce5-8710-699b657febff-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.519066 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.519042 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-metrics-certs" pod "b49683a7-346c-4ce5-8710-699b657febff" (UID: "b49683a7-346c-4ce5-8710-699b657febff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:46.519153 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.519142 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49683a7-346c-4ce5-8710-699b657febff-kube-api-access-xqc4p" pod "b49683a7-346c-4ce5-8710-699b657febff" (UID: "b49683a7-346c-4ce5-8710-699b657febff"). InnerVolumeSpecName "kube-api-access-xqc4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:46.519549 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.519536 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-maas-api-tls" pod "b49683a7-346c-4ce5-8710-699b657febff" (UID: "b49683a7-346c-4ce5-8710-699b657febff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:46.618765 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.618687 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqc4p\" (UniqueName: \"kubernetes.io/projected/b49683a7-346c-4ce5-8710-699b657febff-kube-api-access-xqc4p\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.618765 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.618718 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.618765 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.618728 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49683a7-346c-4ce5-8710-699b657febff-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:46.651149 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.651114 4314 generic.go:356] "Generic (PLEG): container finished" podID="b49683a7-346c-4ce5-8710-699b657febff" containerID="c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657" exitCode=0 Nov 03 20:05:46.651325 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.651158 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" Nov 03 20:05:46.651325 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.651197 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" event={"ID":"b49683a7-346c-4ce5-8710-699b657febff","Type":"ContainerDied","Data":"c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657"} Nov 03 20:05:46.651325 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.651255 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7" event={"ID":"b49683a7-346c-4ce5-8710-699b657febff","Type":"ContainerDied","Data":"3abf12695219fc1b7004ef0266e226bbc0f073a6e3984fb345798dfe001c64b4"} Nov 03 20:05:46.651325 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.651283 4314 scope.go:122] "RemoveContainer" containerID="c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657" Nov 03 20:05:46.698976 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.697723 4314 scope.go:122] "RemoveContainer" containerID="c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657" Nov 03 20:05:46.698976 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:46.698162 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657\": container with ID starting with c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657 not found: ID does not exist" containerID="c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657" Nov 03 20:05:46.698976 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.698202 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657"} err="failed to get container status \"c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657\": rpc error: code = NotFound desc = could not find container \"c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657\": container with ID starting with c0f88e56c627617fc967f994cc962864a47865aa19d3f1beaeeeaecd2c767657 not found: ID does not exist" Nov 03 20:05:46.715763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.715725 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7"] Nov 03 20:05:46.719757 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:46.719734 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-39043d-5f5ff5cffc-tgjp7"] Nov 03 20:05:47.014530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:47.014499 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:05:47.014941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:47.014566 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:05:48.014271 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:48.014234 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:05:48.014483 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:48.014308 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:05:48.428333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:48.428257 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49683a7-346c-4ce5-8710-699b657febff" path="/var/lib/kubelet/pods/b49683a7-346c-4ce5-8710-699b657febff/volumes" Nov 03 20:05:48.485606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:48.482662 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:05:49.013943 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:49.013909 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:05:49.014094 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:49.013973 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:05:49.179506 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:49.179115 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" podUID="f838cc11-3e00-41af-9a21-661befef36e3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:05:49.188745 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:49.188263 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" podUID="8e55db36-642a-4799-a476-a15360e67526" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:05:49.935984 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:49.935945 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" podUID="4a19d0b0-f577-4d5b-b503-22132c13e6ed" containerName="payload-processing" containerID="cri-o://19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb" gracePeriod=25 Nov 03 20:05:50.014124 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.014088 4314 patch_prober.go:29] interesting pod/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:05:50.014292 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.014142 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:05:50.170744 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:05:50.170711 4314 logging.go:55] [core] [Channel #3930 SubChannel #3931]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:50052", ServerName: "10.134.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.48:50052: connect: connection refused" Nov 03 20:05:50.426838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.426803 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 20:05:50.480670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.480637 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a19d0b0-f577-4d5b-b503-22132c13e6ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a19d0b0-f577-4d5b-b503-22132c13e6ed-plugins-config-volume\") pod \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " Nov 03 20:05:50.480896 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.480882 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a19d0b0-f577-4d5b-b503-22132c13e6ed-kube-api-access-mt445\" (UniqueName: \"kubernetes.io/projected/4a19d0b0-f577-4d5b-b503-22132c13e6ed-kube-api-access-mt445\") pod \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\" (UID: \"4a19d0b0-f577-4d5b-b503-22132c13e6ed\") " Nov 03 20:05:50.481719 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.481675 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a19d0b0-f577-4d5b-b503-22132c13e6ed-plugins-config-volume" pod "4a19d0b0-f577-4d5b-b503-22132c13e6ed" (UID: "4a19d0b0-f577-4d5b-b503-22132c13e6ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:50.484009 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.483976 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a19d0b0-f577-4d5b-b503-22132c13e6ed-kube-api-access-mt445" pod "4a19d0b0-f577-4d5b-b503-22132c13e6ed" (UID: "4a19d0b0-f577-4d5b-b503-22132c13e6ed"). InnerVolumeSpecName "kube-api-access-mt445". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:50.583095 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.583047 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a19d0b0-f577-4d5b-b503-22132c13e6ed-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:50.583095 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.583093 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt445\" (UniqueName: \"kubernetes.io/projected/4a19d0b0-f577-4d5b-b503-22132c13e6ed-kube-api-access-mt445\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:50.765429 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.765175 4314 generic.go:356] "Generic (PLEG): container finished" podID="4a19d0b0-f577-4d5b-b503-22132c13e6ed" containerID="19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb" exitCode=0 Nov 03 20:05:50.765429 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.765350 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" event={"ID":"4a19d0b0-f577-4d5b-b503-22132c13e6ed","Type":"ContainerDied","Data":"19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb"} Nov 03 20:05:50.765429 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.765398 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" event={"ID":"4a19d0b0-f577-4d5b-b503-22132c13e6ed","Type":"ContainerDied","Data":"e1080740cfdf042ba849bd1c1e5c9d0cfe74661fb3d5970d6f0ce946882b761b"} Nov 03 20:05:50.765429 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.765426 4314 scope.go:122] "RemoveContainer" containerID="19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb" Nov 03 20:05:50.765845 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.765570 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" Nov 03 20:05:50.795038 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.794915 4314 scope.go:122] "RemoveContainer" containerID="19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb" Nov 03 20:05:50.795515 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:50.795483 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb\": container with ID starting with 19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb not found: ID does not exist" containerID="19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb" Nov 03 20:05:50.795636 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.795529 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb"} err="failed to get container status \"19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb\": rpc error: code = NotFound desc = could not find container \"19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb\": container with ID starting with 19deb84892b948b2e4de525a887527bc9eabbfc863dc9669b74a7bc33693ebfb not found: ID does not exist" Nov 03 20:05:50.808426 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.808402 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq"] Nov 03 20:05:50.817078 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:50.817051 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq"] Nov 03 20:05:51.019207 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.019127 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:05:51.020347 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.020324 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:05:51.171447 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.171404 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a35658-7b7d7b89cc-n97vq" podUID="4a19d0b0-f577-4d5b-b503-22132c13e6ed" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.48:50052\" within 1s: context deadline exceeded" Nov 03 20:05:51.171447 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:05:51.171433 4314 logging.go:55] [core] [Channel #3930 SubChannel #3931]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.48:50052", ServerName: "10.134.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.48:50052: operation was canceled" Nov 03 20:05:51.621796 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.621762 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl"] Nov 03 20:05:51.622227 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.622194 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" containerID="cri-o://60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb" gracePeriod=30 Nov 03 20:05:51.661333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.657176 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 20:05:51.854162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.851327 4314 generic.go:356] "Generic (PLEG): container finished" podID="f838cc11-3e00-41af-9a21-661befef36e3" containerID="211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3" exitCode=0 Nov 03 20:05:51.854162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.851701 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" Nov 03 20:05:51.854162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.851714 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" event={"ID":"f838cc11-3e00-41af-9a21-661befef36e3","Type":"ContainerDied","Data":"211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3"} Nov 03 20:05:51.854162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.851771 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn" event={"ID":"f838cc11-3e00-41af-9a21-661befef36e3","Type":"ContainerDied","Data":"b6567ae04933db7780d3dd4f1f79ab4ff7358ea2d107bd1a0441abb8f164ced3"} Nov 03 20:05:51.854162 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.851800 4314 scope.go:122] "RemoveContainer" containerID="211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3" Nov 03 20:05:51.861148 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.860785 4314 generic.go:356] "Generic (PLEG): container finished" podID="8e55db36-642a-4799-a476-a15360e67526" containerID="7c03cf39c1419594627414ed248977e9e350fd75f75058083334807c2a5361d2" exitCode=0 Nov 03 20:05:51.862084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.862059 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" event={"ID":"8e55db36-642a-4799-a476-a15360e67526","Type":"ContainerDied","Data":"7c03cf39c1419594627414ed248977e9e350fd75f75058083334807c2a5361d2"} Nov 03 20:05:51.862239 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.862220 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" event={"ID":"8e55db36-642a-4799-a476-a15360e67526","Type":"ContainerDied","Data":"ad729c3f6b6c5b224e82606ee82d97e33d91f0ad9ff78af91d2ac9bee9a39b76"} Nov 03 20:05:51.862394 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.862376 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad729c3f6b6c5b224e82606ee82d97e33d91f0ad9ff78af91d2ac9bee9a39b76" Nov 03 20:05:51.868523 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.868033 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 20:05:51.881515 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.881252 4314 scope.go:122] "RemoveContainer" containerID="211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3" Nov 03 20:05:51.881920 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:51.881827 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3\": container with ID starting with 211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3 not found: ID does not exist" containerID="211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3" Nov 03 20:05:51.881920 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.881866 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3"} err="failed to get container status \"211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3\": rpc error: code = NotFound desc = could not find container \"211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3\": container with ID starting with 211cf0a4052bbd049102648d0d843c8b83ba0e53876d7d90ea8797e36ee303b3 not found: ID does not exist" Nov 03 20:05:51.970530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.970481 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e55db36-642a-4799-a476-a15360e67526-kube-api-access-bnksh\" (UniqueName: \"kubernetes.io/projected/8e55db36-642a-4799-a476-a15360e67526-kube-api-access-bnksh\") pod \"8e55db36-642a-4799-a476-a15360e67526\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " Nov 03 20:05:51.970879 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.970853 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f838cc11-3e00-41af-9a21-661befef36e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838cc11-3e00-41af-9a21-661befef36e3-plugins-config-volume\") pod \"f838cc11-3e00-41af-9a21-661befef36e3\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " Nov 03 20:05:51.971153 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.971133 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f838cc11-3e00-41af-9a21-661befef36e3-kube-api-access-9fljk\" (UniqueName: \"kubernetes.io/projected/f838cc11-3e00-41af-9a21-661befef36e3-kube-api-access-9fljk\") pod \"f838cc11-3e00-41af-9a21-661befef36e3\" (UID: \"f838cc11-3e00-41af-9a21-661befef36e3\") " Nov 03 20:05:51.972268 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.971301 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e55db36-642a-4799-a476-a15360e67526-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55db36-642a-4799-a476-a15360e67526-plugins-config-volume\") pod \"8e55db36-642a-4799-a476-a15360e67526\" (UID: \"8e55db36-642a-4799-a476-a15360e67526\") " Nov 03 20:05:51.981671 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.976797 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f838cc11-3e00-41af-9a21-661befef36e3-plugins-config-volume" pod "f838cc11-3e00-41af-9a21-661befef36e3" (UID: "f838cc11-3e00-41af-9a21-661befef36e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:51.982900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.982877 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e55db36-642a-4799-a476-a15360e67526-plugins-config-volume" pod "8e55db36-642a-4799-a476-a15360e67526" (UID: "8e55db36-642a-4799-a476-a15360e67526"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:51.988769 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:51.988702 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f838cc11-3e00-41af-9a21-661befef36e3-kube-api-access-9fljk" pod "f838cc11-3e00-41af-9a21-661befef36e3" (UID: "f838cc11-3e00-41af-9a21-661befef36e3"). InnerVolumeSpecName "kube-api-access-9fljk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:52.037860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.037800 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e55db36-642a-4799-a476-a15360e67526-kube-api-access-bnksh" pod "8e55db36-642a-4799-a476-a15360e67526" (UID: "8e55db36-642a-4799-a476-a15360e67526"). InnerVolumeSpecName "kube-api-access-bnksh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:52.054158 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.054066 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4"] Nov 03 20:05:52.080609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.075334 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fljk\" (UniqueName: \"kubernetes.io/projected/f838cc11-3e00-41af-9a21-661befef36e3-kube-api-access-9fljk\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:52.080609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.075370 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e55db36-642a-4799-a476-a15360e67526-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:52.080609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.075385 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnksh\" (UniqueName: \"kubernetes.io/projected/8e55db36-642a-4799-a476-a15360e67526-kube-api-access-bnksh\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:52.080609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.075399 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f838cc11-3e00-41af-9a21-661befef36e3-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:52.085213 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:52.085082 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:05:52.086924 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.086896 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-742f6235-djzb4"] Nov 03 20:05:52.143355 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143285 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx"] Nov 03 20:05:52.143888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143874 4314 cpu_manager.go:414] "containerMap: removing container" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143889 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143900 4314 cpu_manager.go:414] "containerMap: removing container" podUID="8e55db36-642a-4799-a476-a15360e67526" containerName="payload-pre-processing" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143906 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e55db36-642a-4799-a476-a15360e67526" containerName="payload-pre-processing" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143916 4314 cpu_manager.go:414] "containerMap: removing container" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143921 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143930 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143934 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143941 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4a19d0b0-f577-4d5b-b503-22132c13e6ed" containerName="payload-processing" Nov 03 20:05:52.143944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143945 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a19d0b0-f577-4d5b-b503-22132c13e6ed" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143954 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b49683a7-346c-4ce5-8710-699b657febff" containerName="maas-api" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143960 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49683a7-346c-4ce5-8710-699b657febff" containerName="maas-api" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143978 4314 cpu_manager.go:414] "containerMap: removing container" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143982 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143994 4314 cpu_manager.go:414] "containerMap: removing container" podUID="e70b7ba4-f07c-4b53-89b9-80bb29186ad3" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.143998 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70b7ba4-f07c-4b53-89b9-80bb29186ad3" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144010 4314 cpu_manager.go:414] "containerMap: removing container" podUID="d4007995-7031-440b-ba72-08761b3fd345" containerName="revoke-keys" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144015 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4007995-7031-440b-ba72-08761b3fd345" containerName="revoke-keys" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144032 4314 cpu_manager.go:414] "containerMap: removing container" podUID="f838cc11-3e00-41af-9a21-661befef36e3" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144036 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f838cc11-3e00-41af-9a21-661befef36e3" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144123 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f838cc11-3e00-41af-9a21-661befef36e3" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144137 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad274c2-4168-4b70-a353-52475635630d" containerName="main" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144144 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e55db36-642a-4799-a476-a15360e67526" containerName="payload-pre-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144150 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144156 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb057b6-7660-4a8e-9cd5-3ffe086cd72d" containerName="main" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144161 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a19d0b0-f577-4d5b-b503-22132c13e6ed" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144166 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49683a7-346c-4ce5-8710-699b657febff" containerName="maas-api" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144172 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bf6477-c859-4d11-8de8-aff72c7d849d" containerName="main" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144179 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70b7ba4-f07c-4b53-89b9-80bb29186ad3" containerName="payload-processing" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144184 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4007995-7031-440b-ba72-08761b3fd345" containerName="revoke-keys" Nov 03 20:05:52.144224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.144191 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c956c314-2fcc-4302-a6ac-11834c80cf9a" containerName="istio-proxy" Nov 03 20:05:52.147157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.147138 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.151180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.151160 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:05:52.151359 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.151347 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:05:52.156050 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.154421 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx"] Nov 03 20:05:52.191628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.180803 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85fr\" (UniqueName: \"kubernetes.io/projected/fdf160b8-8393-4590-95f7-f59769271c0b-kube-api-access-l85fr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.191628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.180865 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.191628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.180912 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.256572 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.256546 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn"] Nov 03 20:05:52.306804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.293286 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85fr\" (UniqueName: \"kubernetes.io/projected/fdf160b8-8393-4590-95f7-f59769271c0b-kube-api-access-l85fr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.306804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.293356 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.306804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.293404 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.306804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.294350 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.330999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.330230 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-39043d-5dbd77d679-94xqn"] Nov 03 20:05:52.330999 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.330261 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.333619 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.333595 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85fr\" (UniqueName: \"kubernetes.io/projected/fdf160b8-8393-4590-95f7-f59769271c0b-kube-api-access-l85fr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.430990 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.430904 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a19d0b0-f577-4d5b-b503-22132c13e6ed" path="/var/lib/kubelet/pods/4a19d0b0-f577-4d5b-b503-22132c13e6ed/volumes" Nov 03 20:05:52.431452 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.431434 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f838cc11-3e00-41af-9a21-661befef36e3" path="/var/lib/kubelet/pods/f838cc11-3e00-41af-9a21-661befef36e3/volumes" Nov 03 20:05:52.431917 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.431901 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea4de96-2c62-4fa9-befc-a03e630b88fd" path="/var/lib/kubelet/pods/fea4de96-2c62-4fa9-befc-a03e630b88fd/volumes" Nov 03 20:05:52.538447 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.538415 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:52.826377 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.826350 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx"] Nov 03 20:05:52.827447 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:05:52.827424 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf160b8_8393_4590_95f7_f59769271c0b.slice/crio-31680596965c078b8fbfd91ceb509be2e373004217c679089e2e1dcc81fb3df9 WatchSource:0}: Error finding container 31680596965c078b8fbfd91ceb509be2e373004217c679089e2e1dcc81fb3df9: Status 404 returned error can't find the container with id 31680596965c078b8fbfd91ceb509be2e373004217c679089e2e1dcc81fb3df9 Nov 03 20:05:52.884373 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.884330 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" event={"ID":"fdf160b8-8393-4590-95f7-f59769271c0b","Type":"ContainerStarted","Data":"31680596965c078b8fbfd91ceb509be2e373004217c679089e2e1dcc81fb3df9"} Nov 03 20:05:52.885628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.885604 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z" Nov 03 20:05:52.904461 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.904430 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z"] Nov 03 20:05:52.910924 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.910899 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-39043d-66d86796c5-9725z"] Nov 03 20:05:52.973800 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.973767 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r"] Nov 03 20:05:52.974282 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.974248 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" containerID="cri-o://5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c" gracePeriod=30 Nov 03 20:05:52.979458 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.979435 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn"] Nov 03 20:05:52.980114 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:52.980045 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" containerID="cri-o://137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66" gracePeriod=30 Nov 03 20:05:53.893849 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:53.893820 4314 generic.go:356] "Generic (PLEG): container finished" podID="fdf160b8-8393-4590-95f7-f59769271c0b" containerID="38a0f24c24bd9b8cce2b817afe0fa230fe3276185500f4a98d437264e1a48761" exitCode=0 Nov 03 20:05:53.894216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:53.893915 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" event={"ID":"fdf160b8-8393-4590-95f7-f59769271c0b","Type":"ContainerDied","Data":"38a0f24c24bd9b8cce2b817afe0fa230fe3276185500f4a98d437264e1a48761"} Nov 03 20:05:54.441614 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.441423 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e55db36-642a-4799-a476-a15360e67526" path="/var/lib/kubelet/pods/8e55db36-642a-4799-a476-a15360e67526/volumes" Nov 03 20:05:54.623989 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.623944 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" containerID="cri-o://494bf993d36d70742affa9b932b6232b310945e9c6d774c9aa44cd8afb3949bf" gracePeriod=45 Nov 03 20:05:54.644124 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.644086 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" containerID="cri-o://c5596541f038c6c7e2d26a43d71b947eb83e4581ea7a43338cbece418936489e" gracePeriod=45 Nov 03 20:05:54.720148 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:54.720086 4314 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d337f6_4a62_4fa7_a6ec_25f562474a8d.slice/crio-c5596541f038c6c7e2d26a43d71b947eb83e4581ea7a43338cbece418936489e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:05:54.919525 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.919487 4314 generic.go:356] "Generic (PLEG): container finished" podID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerID="494bf993d36d70742affa9b932b6232b310945e9c6d774c9aa44cd8afb3949bf" exitCode=0 Nov 03 20:05:54.920266 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.919652 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" event={"ID":"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c","Type":"ContainerDied","Data":"494bf993d36d70742affa9b932b6232b310945e9c6d774c9aa44cd8afb3949bf"} Nov 03 20:05:54.921924 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.921896 4314 generic.go:356] "Generic (PLEG): container finished" podID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerID="c5596541f038c6c7e2d26a43d71b947eb83e4581ea7a43338cbece418936489e" exitCode=0 Nov 03 20:05:54.922254 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.922228 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" event={"ID":"18d337f6-4a62-4fa7-a6ec-25f562474a8d","Type":"ContainerDied","Data":"c5596541f038c6c7e2d26a43d71b947eb83e4581ea7a43338cbece418936489e"} Nov 03 20:05:54.948094 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.948068 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 20:05:55.001627 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:54.999676 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 20:05:55.059929 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.059892 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-model-cache\") pod \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " Nov 03 20:05:55.060212 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.060194 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-dshm\") pod \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " Nov 03 20:05:55.060486 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.060471 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-kube-api-access-7dlvl\" (UniqueName: \"kubernetes.io/projected/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-kube-api-access-7dlvl\") pod \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " Nov 03 20:05:55.061230 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.061196 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-home\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-home\") pod \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " Nov 03 20:05:55.061373 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.061333 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-dshm\") pod \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " Nov 03 20:05:55.061436 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.061376 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tls-certs\") pod \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " Nov 03 20:05:55.061436 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.061413 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-home\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-home\") pod \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " Nov 03 20:05:55.061547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.061494 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tmp-dir\") pod \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\" (UID: \"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c\") " Nov 03 20:05:55.062233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.062207 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-model-cache" pod "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" (UID: "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.062440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.062416 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-home" pod "18d337f6-4a62-4fa7-a6ec-25f562474a8d" (UID: "18d337f6-4a62-4fa7-a6ec-25f562474a8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.062786 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.062763 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tmp-dir" pod "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" (UID: "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.063022 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.063001 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-home" pod "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" (UID: "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.066060 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.066019 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-kube-api-access-7dlvl" pod "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" (UID: "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c"). InnerVolumeSpecName "kube-api-access-7dlvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:55.066315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.066297 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-dshm" pod "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" (UID: "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.066761 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.066725 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-dshm" pod "18d337f6-4a62-4fa7-a6ec-25f562474a8d" (UID: "18d337f6-4a62-4fa7-a6ec-25f562474a8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.068522 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.068487 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tls-certs" pod "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" (UID: "8ff0d55c-5ae2-45be-8edb-d770f7ffb20c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:55.154293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.154268 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:55.162453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.162424 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-openshift-ca\") pod \"fdf160b8-8393-4590-95f7-f59769271c0b\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " Nov 03 20:05:55.162643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.162501 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-model-cache\") pod \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " Nov 03 20:05:55.162643 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.162553 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tls-certs\") pod \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " Nov 03 20:05:55.162783 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.162750 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-model-cache" pod "18d337f6-4a62-4fa7-a6ec-25f562474a8d" (UID: "18d337f6-4a62-4fa7-a6ec-25f562474a8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.162783 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.162755 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-openshift-ca" pod "fdf160b8-8393-4590-95f7-f59769271c0b" (UID: "fdf160b8-8393-4590-95f7-f59769271c0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:55.163009 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.162989 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tmp-dir\") pod \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " Nov 03 20:05:55.163066 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163034 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf160b8-8393-4590-95f7-f59769271c0b-kube-api-access-l85fr\" (UniqueName: \"kubernetes.io/projected/fdf160b8-8393-4590-95f7-f59769271c0b-kube-api-access-l85fr\") pod \"fdf160b8-8393-4590-95f7-f59769271c0b\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " Nov 03 20:05:55.163128 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163104 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-xks-ca\") pod \"fdf160b8-8393-4590-95f7-f59769271c0b\" (UID: \"fdf160b8-8393-4590-95f7-f59769271c0b\") " Nov 03 20:05:55.163227 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163211 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d337f6-4a62-4fa7-a6ec-25f562474a8d-kube-api-access-247xq\" (UniqueName: \"kubernetes.io/projected/18d337f6-4a62-4fa7-a6ec-25f562474a8d-kube-api-access-247xq\") pod \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\" (UID: \"18d337f6-4a62-4fa7-a6ec-25f562474a8d\") " Nov 03 20:05:55.163288 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163231 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tmp-dir" pod "18d337f6-4a62-4fa7-a6ec-25f562474a8d" (UID: "18d337f6-4a62-4fa7-a6ec-25f562474a8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:55.163838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163808 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.163838 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163837 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163851 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163864 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163877 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlvl\" (UniqueName: \"kubernetes.io/projected/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-kube-api-access-7dlvl\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163890 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163903 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d337f6-4a62-4fa7-a6ec-25f562474a8d-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163915 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163930 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163942 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.164003 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.163955 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.165257 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.165227 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tls-certs" pod "18d337f6-4a62-4fa7-a6ec-25f562474a8d" (UID: "18d337f6-4a62-4fa7-a6ec-25f562474a8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:55.165518 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.165501 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d337f6-4a62-4fa7-a6ec-25f562474a8d-kube-api-access-247xq" pod "18d337f6-4a62-4fa7-a6ec-25f562474a8d" (UID: "18d337f6-4a62-4fa7-a6ec-25f562474a8d"). InnerVolumeSpecName "kube-api-access-247xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:55.165616 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.165527 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf160b8-8393-4590-95f7-f59769271c0b-kube-api-access-l85fr" pod "fdf160b8-8393-4590-95f7-f59769271c0b" (UID: "fdf160b8-8393-4590-95f7-f59769271c0b"). InnerVolumeSpecName "kube-api-access-l85fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:55.166002 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.165987 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-xks-ca" pod "fdf160b8-8393-4590-95f7-f59769271c0b" (UID: "fdf160b8-8393-4590-95f7-f59769271c0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:55.265134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.265097 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247xq\" (UniqueName: \"kubernetes.io/projected/18d337f6-4a62-4fa7-a6ec-25f562474a8d-kube-api-access-247xq\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.265134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.265138 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d337f6-4a62-4fa7-a6ec-25f562474a8d-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.265409 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.265153 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l85fr\" (UniqueName: \"kubernetes.io/projected/fdf160b8-8393-4590-95f7-f59769271c0b-kube-api-access-l85fr\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.265409 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.265167 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf160b8-8393-4590-95f7-f59769271c0b-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:55.574561 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.574475 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt"] Nov 03 20:05:55.932315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.932224 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" Nov 03 20:05:55.932315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.932247 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx" event={"ID":"fdf160b8-8393-4590-95f7-f59769271c0b","Type":"ContainerDied","Data":"31680596965c078b8fbfd91ceb509be2e373004217c679089e2e1dcc81fb3df9"} Nov 03 20:05:55.932315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.932300 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31680596965c078b8fbfd91ceb509be2e373004217c679089e2e1dcc81fb3df9" Nov 03 20:05:55.934011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.933978 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" event={"ID":"8ff0d55c-5ae2-45be-8edb-d770f7ffb20c","Type":"ContainerDied","Data":"0234943c6e531e900394dede9d561d2ae456393c4eb5916c8780688e76b15c3d"} Nov 03 20:05:55.934152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.934075 4314 scope.go:122] "RemoveContainer" containerID="494bf993d36d70742affa9b932b6232b310945e9c6d774c9aa44cd8afb3949bf" Nov 03 20:05:55.934152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.933982 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff" Nov 03 20:05:55.936742 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.936723 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" Nov 03 20:05:55.936828 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.936751 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4" event={"ID":"18d337f6-4a62-4fa7-a6ec-25f562474a8d","Type":"ContainerDied","Data":"3c8865989bb0ebff548fdf74c520413e087f71d21f00701f1fb2adfe414d17fd"} Nov 03 20:05:55.947032 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.946895 4314 scope.go:122] "RemoveContainer" containerID="c5596541f038c6c7e2d26a43d71b947eb83e4581ea7a43338cbece418936489e" Nov 03 20:05:55.970612 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.970556 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4"] Nov 03 20:05:55.974222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.974192 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/simulator-w1-a35658-kserve-5d4df55d8b-flst4"] Nov 03 20:05:55.981934 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.981907 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff"] Nov 03 20:05:55.984703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:55.984683 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-2-w1-a35658-kserve-7d4cdb8f8f-v6pff"] Nov 03 20:05:56.429304 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.429269 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" path="/var/lib/kubelet/pods/18d337f6-4a62-4fa7-a6ec-25f562474a8d/volumes" Nov 03 20:05:56.429783 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.429768 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" path="/var/lib/kubelet/pods/8ff0d55c-5ae2-45be-8edb-d770f7ffb20c/volumes" Nov 03 20:05:56.544379 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.544339 4314 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:05:56.544548 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.544402 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:05:56.952735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.952262 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 20:05:56.960037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.960008 4314 generic.go:356] "Generic (PLEG): container finished" podID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerID="60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb" exitCode=0 Nov 03 20:05:56.960198 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.960060 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" Nov 03 20:05:56.960198 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.960090 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" event={"ID":"7eb09d7b-edcd-4f5b-a297-b700f4b32b59","Type":"ContainerDied","Data":"60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb"} Nov 03 20:05:56.960198 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.960142 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl" event={"ID":"7eb09d7b-edcd-4f5b-a297-b700f4b32b59","Type":"ContainerDied","Data":"52f6fc3b0d265a4d21ce391011cd6a45ccd4e312114dffd854789cdca63e25df"} Nov 03 20:05:56.960198 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.960174 4314 scope.go:122] "RemoveContainer" containerID="60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb" Nov 03 20:05:56.971055 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.971037 4314 scope.go:122] "RemoveContainer" containerID="c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d" Nov 03 20:05:56.983820 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.983793 4314 scope.go:122] "RemoveContainer" containerID="60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb" Nov 03 20:05:56.984116 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:56.984093 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb\": container with ID starting with 60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb not found: ID does not exist" containerID="60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb" Nov 03 20:05:56.984235 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.984128 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb"} err="failed to get container status \"60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb\": rpc error: code = NotFound desc = could not find container \"60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb\": container with ID starting with 60beedbeeb21d144722143bb7acf5fa1812b61a74487522147035a7362ed6ccb not found: ID does not exist" Nov 03 20:05:56.984235 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.984151 4314 scope.go:122] "RemoveContainer" containerID="c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d" Nov 03 20:05:56.984452 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:56.984427 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d\": container with ID starting with c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d not found: ID does not exist" containerID="c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d" Nov 03 20:05:56.984565 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.984456 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d"} err="failed to get container status \"c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d\": rpc error: code = NotFound desc = could not find container \"c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d\": container with ID starting with c5ec95a5fc51a48e7ed9525e1e015725328f317866bd1160211fdd8034fb021d not found: ID does not exist" Nov 03 20:05:56.992599 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.992556 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-credential-socket\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.992691 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.992646 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-socket\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.992796 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.992780 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-certs\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.992867 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.992852 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istiod-ca-cert\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.992927 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.992903 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-socket" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:56.992971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.992917 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-data\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.993028 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993011 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-kube-api-access-pd6sm\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-kube-api-access-pd6sm\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.993094 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993058 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-token\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-token\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.993148 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993125 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-data" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:56.993148 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993141 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-podinfo\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.993233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993212 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-envoy\") pod \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\" (UID: \"7eb09d7b-edcd-4f5b-a297-b700f4b32b59\") " Nov 03 20:05:56.993389 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993369 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-certs" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:56.993757 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993729 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istiod-ca-cert" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:56.993836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993817 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:56.993876 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993843 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:56.993876 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.993858 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:56.994639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.994614 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-credential-socket" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:56.996111 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.996003 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-token" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:56.996522 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.996498 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-podinfo" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:56.996819 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.996794 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-kube-api-access-pd6sm" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "kube-api-access-pd6sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:56.997415 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:56.997391 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-envoy" pod "7eb09d7b-edcd-4f5b-a297-b700f4b32b59" (UID: "7eb09d7b-edcd-4f5b-a297-b700f4b32b59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:57.095205 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.095176 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:57.095205 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.095200 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd6sm\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-kube-api-access-pd6sm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:57.095421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.095209 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:57.095421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.095232 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:57.095421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.095242 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:57.095421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.095251 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eb09d7b-edcd-4f5b-a297-b700f4b32b59-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:57.290631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.290573 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl"] Nov 03 20:05:57.298735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:57.298709 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c18bea-openshift-default-5f44c48cbf-zs9fl"] Nov 03 20:05:58.336476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.333859 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.424167 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-kube-api-access-mpzcd\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-kube-api-access-mpzcd\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.424234 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-envoy\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.424345 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-data\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.424447 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-podinfo\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.424487 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-socket\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.424518 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-token\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-token\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.424973 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-data" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.426109 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-socket" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.426423 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-certs\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.426513 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istiod-ca-cert\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.426562 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-credential-socket\") pod \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\" (UID: \"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1\") " Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.427451 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-certs" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.427750 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istiod-ca-cert" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:58.431187 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.427804 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-credential-socket" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.442265 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.442221 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" path="/var/lib/kubelet/pods/7eb09d7b-edcd-4f5b-a297-b700f4b32b59/volumes" Nov 03 20:05:58.449400 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.449199 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-podinfo" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:58.451335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.450737 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-envoy" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.451335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.451300 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-token" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:58.456049 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.456011 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-kube-api-access-mpzcd" pod "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" (UID: "22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1"). InnerVolumeSpecName "kube-api-access-mpzcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528573 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528619 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528637 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpzcd\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-kube-api-access-mpzcd\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528658 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528673 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528687 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528701 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528714 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.528746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.528726 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.563651 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.563620 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 20:05:58.629696 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.629661 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-plugins-config-volume\") pod \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " Nov 03 20:05:58.629863 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.629843 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-kube-api-access-7vtmr\" (UniqueName: \"kubernetes.io/projected/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-kube-api-access-7vtmr\") pod \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\" (UID: \"5cc63876-7c5c-4a54-8227-d78f36e4e4d9\") " Nov 03 20:05:58.630083 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.630058 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-plugins-config-volume" pod "5cc63876-7c5c-4a54-8227-d78f36e4e4d9" (UID: "5cc63876-7c5c-4a54-8227-d78f36e4e4d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:58.630228 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.630207 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.632844 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.632817 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-kube-api-access-7vtmr" pod "5cc63876-7c5c-4a54-8227-d78f36e4e4d9" (UID: "5cc63876-7c5c-4a54-8227-d78f36e4e4d9"). InnerVolumeSpecName "kube-api-access-7vtmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:58.661335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.661312 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 20:05:58.731401 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731369 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-credential-socket\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.731652 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731477 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-podinfo\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.731652 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731617 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-envoy\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.731799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731655 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-token\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-token\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.731799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731656 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-credential-socket" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.731799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731696 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-socket\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.731799 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731757 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-certs\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.732064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731796 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-data\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.732064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731882 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istiod-ca-cert\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.732064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.731934 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-kube-api-access-ss5vh\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-kube-api-access-ss5vh\") pod \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\" (UID: \"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5\") " Nov 03 20:05:58.732243 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.732218 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-socket" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.732758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.732599 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vtmr\" (UniqueName: \"kubernetes.io/projected/5cc63876-7c5c-4a54-8227-d78f36e4e4d9-kube-api-access-7vtmr\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.732758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.732627 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.732758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.732679 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.733668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.732893 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-certs" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.733668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.733006 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istiod-ca-cert" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:58.733668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.733075 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-data" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.741334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.741303 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-kube-api-access-ss5vh" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "kube-api-access-ss5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:58.741558 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.741535 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-token" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:58.741661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.741643 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-podinfo" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:05:58.741714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.741701 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-envoy" pod "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" (UID: "0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:05:58.833804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.833719 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.833804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.833760 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.833804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.833775 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.833804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.833790 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss5vh\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-kube-api-access-ss5vh\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.833804 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.833804 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.834072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.833818 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.834072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.833834 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:58.980482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.980448 4314 generic.go:356] "Generic (PLEG): container finished" podID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerID="e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657" exitCode=0 Nov 03 20:05:58.980709 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.980497 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" Nov 03 20:05:58.980709 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.980535 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" event={"ID":"5cc63876-7c5c-4a54-8227-d78f36e4e4d9","Type":"ContainerDied","Data":"e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657"} Nov 03 20:05:58.980709 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.980620 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" event={"ID":"5cc63876-7c5c-4a54-8227-d78f36e4e4d9","Type":"ContainerDied","Data":"b9e8fa2b1173839ce0c143851f4f0c28af33acaadc89b8d1c3e21d5dace08c11"} Nov 03 20:05:58.980709 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.980654 4314 scope.go:122] "RemoveContainer" containerID="e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657" Nov 03 20:05:58.983071 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.983047 4314 generic.go:356] "Generic (PLEG): container finished" podID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerID="5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c" exitCode=0 Nov 03 20:05:58.984456 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.983109 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" Nov 03 20:05:58.984456 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.983119 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" event={"ID":"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5","Type":"ContainerDied","Data":"5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c"} Nov 03 20:05:58.984456 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.983494 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r" event={"ID":"0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5","Type":"ContainerDied","Data":"e93ca494f679381dc1e32b77d1298af6c9deba8e2ef83d8de453d08b62cdf772"} Nov 03 20:05:58.990970 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.990945 4314 generic.go:356] "Generic (PLEG): container finished" podID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerID="137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66" exitCode=0 Nov 03 20:05:58.991092 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.990992 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" Nov 03 20:05:58.991258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.990995 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" event={"ID":"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1","Type":"ContainerDied","Data":"137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66"} Nov 03 20:05:58.991258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.991220 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn" event={"ID":"22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1","Type":"ContainerDied","Data":"07d3584bbb46c42ddd33b619f1c17984adaea7a13b85376fc8e383a844a8bc03"} Nov 03 20:05:58.998496 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.998420 4314 scope.go:122] "RemoveContainer" containerID="e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657" Nov 03 20:05:58.999209 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:58.999181 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657\": container with ID starting with e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657 not found: ID does not exist" containerID="e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657" Nov 03 20:05:58.999298 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.999222 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657"} err="failed to get container status \"e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657\": rpc error: code = NotFound desc = could not find container \"e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657\": container with ID starting with e5a9ef592dc4269f7705b985fd45d830f14a3beb1d9d48f1f0bc1668b63a3657 not found: ID does not exist" Nov 03 20:05:58.999298 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:58.999246 4314 scope.go:122] "RemoveContainer" containerID="5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c" Nov 03 20:05:59.019292 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.019276 4314 scope.go:122] "RemoveContainer" containerID="5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c" Nov 03 20:05:59.019559 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:59.019542 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c\": container with ID starting with 5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c not found: ID does not exist" containerID="5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c" Nov 03 20:05:59.019628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.019566 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c"} err="failed to get container status \"5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c\": rpc error: code = NotFound desc = could not find container \"5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c\": container with ID starting with 5317b68908db4af447ff9ecfbe892473fc6ff6912c0cbf79d969063cb794d69c not found: ID does not exist" Nov 03 20:05:59.019628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.019594 4314 scope.go:122] "RemoveContainer" containerID="137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66" Nov 03 20:05:59.028815 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.028795 4314 scope.go:122] "RemoveContainer" containerID="5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d" Nov 03 20:05:59.033731 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.033713 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs"] Nov 03 20:05:59.037023 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.037001 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs"] Nov 03 20:05:59.038858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.038840 4314 scope.go:122] "RemoveContainer" containerID="137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66" Nov 03 20:05:59.039113 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:59.039093 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66\": container with ID starting with 137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66 not found: ID does not exist" containerID="137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66" Nov 03 20:05:59.039189 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.039124 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66"} err="failed to get container status \"137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66\": rpc error: code = NotFound desc = could not find container \"137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66\": container with ID starting with 137bbe24a59f12f649e3a767d695f2774d69e58c8f9d6795bb06915636f5bc66 not found: ID does not exist" Nov 03 20:05:59.039189 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.039145 4314 scope.go:122] "RemoveContainer" containerID="5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d" Nov 03 20:05:59.039395 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:05:59.039378 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d\": container with ID starting with 5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d not found: ID does not exist" containerID="5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d" Nov 03 20:05:59.039440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.039401 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d"} err="failed to get container status \"5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d\": rpc error: code = NotFound desc = could not find container \"5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d\": container with ID starting with 5c017805226e98b89544f77de8d7bdac2fd00febb97205ab1ca00c5c85bdff6d not found: ID does not exist" Nov 03 20:05:59.050200 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.050181 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r"] Nov 03 20:05:59.056348 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.056317 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-9qs8r"] Nov 03 20:05:59.071360 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.071330 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn"] Nov 03 20:05:59.079475 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.079449 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-hp9dn"] Nov 03 20:05:59.467689 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:05:59.467635 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c18bea-767ccf7f75-n8xvs" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.126:50052\" within 1s: context deadline exceeded" Nov 03 20:05:59.467689 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:05:59.467680 4314 logging.go:55] [core] [Channel #3944 SubChannel #3945]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.126:50052", ServerName: "10.134.0.126:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.126:50052: operation was canceled" Nov 03 20:06:00.428727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:00.428684 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" path="/var/lib/kubelet/pods/0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5/volumes" Nov 03 20:06:00.429324 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:00.429301 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" path="/var/lib/kubelet/pods/22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1/volumes" Nov 03 20:06:00.429806 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:00.429786 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" path="/var/lib/kubelet/pods/5cc63876-7c5c-4a54-8227-d78f36e4e4d9/volumes" Nov 03 20:06:02.281323 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:02.281278 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n"] Nov 03 20:06:06.386173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:06.386125 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx"] Nov 03 20:06:06.389672 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:06.389636 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-6eb768-k8lmx"] Nov 03 20:06:06.460915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:06.460871 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf160b8-8393-4590-95f7-f59769271c0b" path="/var/lib/kubelet/pods/fdf160b8-8393-4590-95f7-f59769271c0b/volumes" Nov 03 20:06:07.281722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.281651 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" podUID="ad7c86d7-e349-4408-be55-626b637e7db6" containerName="payload-processing" containerID="cri-o://c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa" gracePeriod=25 Nov 03 20:06:07.689013 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.688980 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 20:06:07.785411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.785370 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7c86d7-e349-4408-be55-626b637e7db6-kube-api-access-g8z25\" (UniqueName: \"kubernetes.io/projected/ad7c86d7-e349-4408-be55-626b637e7db6-kube-api-access-g8z25\") pod \"ad7c86d7-e349-4408-be55-626b637e7db6\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " Nov 03 20:06:07.785613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.785536 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7c86d7-e349-4408-be55-626b637e7db6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7c86d7-e349-4408-be55-626b637e7db6-plugins-config-volume\") pod \"ad7c86d7-e349-4408-be55-626b637e7db6\" (UID: \"ad7c86d7-e349-4408-be55-626b637e7db6\") " Nov 03 20:06:07.786623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.786571 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7c86d7-e349-4408-be55-626b637e7db6-plugins-config-volume" pod "ad7c86d7-e349-4408-be55-626b637e7db6" (UID: "ad7c86d7-e349-4408-be55-626b637e7db6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:07.789058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.789024 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7c86d7-e349-4408-be55-626b637e7db6-kube-api-access-g8z25" pod "ad7c86d7-e349-4408-be55-626b637e7db6" (UID: "ad7c86d7-e349-4408-be55-626b637e7db6"). InnerVolumeSpecName "kube-api-access-g8z25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:07.887338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.887190 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7c86d7-e349-4408-be55-626b637e7db6-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:07.887338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:07.887224 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8z25\" (UniqueName: \"kubernetes.io/projected/ad7c86d7-e349-4408-be55-626b637e7db6-kube-api-access-g8z25\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:08.061175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.061090 4314 generic.go:356] "Generic (PLEG): container finished" podID="ad7c86d7-e349-4408-be55-626b637e7db6" containerID="c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa" exitCode=0 Nov 03 20:06:08.061175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.061142 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" Nov 03 20:06:08.061623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.061173 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" event={"ID":"ad7c86d7-e349-4408-be55-626b637e7db6","Type":"ContainerDied","Data":"c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa"} Nov 03 20:06:08.061623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.061224 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n" event={"ID":"ad7c86d7-e349-4408-be55-626b637e7db6","Type":"ContainerDied","Data":"0e5dafe2b4cebdd850e6e424736d6d3574ad354765dd22bcf7b4a98b0da20517"} Nov 03 20:06:08.061623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.061246 4314 scope.go:122] "RemoveContainer" containerID="c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa" Nov 03 20:06:08.073295 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.073273 4314 scope.go:122] "RemoveContainer" containerID="c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa" Nov 03 20:06:08.073620 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:08.073595 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa\": container with ID starting with c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa not found: ID does not exist" containerID="c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa" Nov 03 20:06:08.073675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.073628 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa"} err="failed to get container status \"c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa\": rpc error: code = NotFound desc = could not find container \"c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa\": container with ID starting with c387e4256f7c6e47194771f7fc532fce3cf2b14168f2b6796f45d807bdfe1afa not found: ID does not exist" Nov 03 20:06:08.074043 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.074026 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:06:08.108528 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.108501 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n"] Nov 03 20:06:08.123626 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.123600 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-6eb768-7fd7b5b985-qsb5n"] Nov 03 20:06:08.429156 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:08.429074 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7c86d7-e349-4408-be55-626b637e7db6" path="/var/lib/kubelet/pods/ad7c86d7-e349-4408-be55-626b637e7db6/volumes" Nov 03 20:06:10.595889 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.595846 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" containerID="cri-o://37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20" gracePeriod=45 Nov 03 20:06:10.696230 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.696192 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused" Nov 03 20:06:10.877368 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.877342 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:06:10.929430 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929401 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-home\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-home\") pod \"257d4457-e428-4e46-997d-dbab9754eb5d\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " Nov 03 20:06:10.929632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929447 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-model-cache\") pod \"257d4457-e428-4e46-997d-dbab9754eb5d\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " Nov 03 20:06:10.929632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929493 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257d4457-e428-4e46-997d-dbab9754eb5d-kube-api-access-f8sgd\" (UniqueName: \"kubernetes.io/projected/257d4457-e428-4e46-997d-dbab9754eb5d-kube-api-access-f8sgd\") pod \"257d4457-e428-4e46-997d-dbab9754eb5d\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " Nov 03 20:06:10.929632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929573 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-tmp-dir\") pod \"257d4457-e428-4e46-997d-dbab9754eb5d\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " Nov 03 20:06:10.929899 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929688 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-dshm\") pod \"257d4457-e428-4e46-997d-dbab9754eb5d\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " Nov 03 20:06:10.929899 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929721 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257d4457-e428-4e46-997d-dbab9754eb5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/257d4457-e428-4e46-997d-dbab9754eb5d-tls-certs\") pod \"257d4457-e428-4e46-997d-dbab9754eb5d\" (UID: \"257d4457-e428-4e46-997d-dbab9754eb5d\") " Nov 03 20:06:10.929899 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929775 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-home" pod "257d4457-e428-4e46-997d-dbab9754eb5d" (UID: "257d4457-e428-4e46-997d-dbab9754eb5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:10.929899 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.929786 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-model-cache" pod "257d4457-e428-4e46-997d-dbab9754eb5d" (UID: "257d4457-e428-4e46-997d-dbab9754eb5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:10.930211 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.930176 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-tmp-dir" pod "257d4457-e428-4e46-997d-dbab9754eb5d" (UID: "257d4457-e428-4e46-997d-dbab9754eb5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:10.931624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.930423 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:10.931624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.930447 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:10.931624 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.930461 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:10.933693 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.932614 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-dshm" pod "257d4457-e428-4e46-997d-dbab9754eb5d" (UID: "257d4457-e428-4e46-997d-dbab9754eb5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:10.933693 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.932689 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257d4457-e428-4e46-997d-dbab9754eb5d-tls-certs" pod "257d4457-e428-4e46-997d-dbab9754eb5d" (UID: "257d4457-e428-4e46-997d-dbab9754eb5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:10.933693 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:10.932724 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257d4457-e428-4e46-997d-dbab9754eb5d-kube-api-access-f8sgd" pod "257d4457-e428-4e46-997d-dbab9754eb5d" (UID: "257d4457-e428-4e46-997d-dbab9754eb5d"). InnerVolumeSpecName "kube-api-access-f8sgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:11.031951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.031914 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sgd\" (UniqueName: \"kubernetes.io/projected/257d4457-e428-4e46-997d-dbab9754eb5d-kube-api-access-f8sgd\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:11.031951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.031954 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257d4457-e428-4e46-997d-dbab9754eb5d-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:11.032141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.031973 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257d4457-e428-4e46-997d-dbab9754eb5d-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:11.094940 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.094889 4314 generic.go:356] "Generic (PLEG): container finished" podID="257d4457-e428-4e46-997d-dbab9754eb5d" containerID="37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20" exitCode=0 Nov 03 20:06:11.095148 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.095083 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" event={"ID":"257d4457-e428-4e46-997d-dbab9754eb5d","Type":"ContainerDied","Data":"37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20"} Nov 03 20:06:11.095148 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.095128 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" event={"ID":"257d4457-e428-4e46-997d-dbab9754eb5d","Type":"ContainerDied","Data":"9d4193086995009cc700f13a2a06664cb39b2335b46a4f86a9098c3c1b06c8f0"} Nov 03 20:06:11.095261 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.095165 4314 scope.go:122] "RemoveContainer" containerID="37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20" Nov 03 20:06:11.095334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.095314 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt" Nov 03 20:06:11.113326 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.113305 4314 scope.go:122] "RemoveContainer" containerID="37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20" Nov 03 20:06:11.115077 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:11.115045 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20\": container with ID starting with 37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20 not found: ID does not exist" containerID="37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20" Nov 03 20:06:11.115176 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.115088 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20"} err="failed to get container status \"37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20\": rpc error: code = NotFound desc = could not find container \"37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20\": container with ID starting with 37c903c7ea817d3a57857fc876a6f8f2772e9ab4ccb75514eef996a52082ea20 not found: ID does not exist" Nov 03 20:06:11.153196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.153173 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt"] Nov 03 20:06:11.162647 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:11.162625 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/premium-w2-8d4127-kserve-b9bb7d448-cjhnt"] Nov 03 20:06:12.428424 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:12.428390 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" path="/var/lib/kubelet/pods/257d4457-e428-4e46-997d-dbab9754eb5d/volumes" Nov 03 20:06:14.615063 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:14.615024 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8"] Nov 03 20:06:14.615551 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:14.615296 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" containerID="cri-o://3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8" gracePeriod=30 Nov 03 20:06:19.882013 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.881946 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 20:06:19.980971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.980936 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-certs\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.980971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.980976 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-podinfo\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981000 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-data\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981064 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-token\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-token\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981204 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-envoy\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981248 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-credential-socket\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981281 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1abade-07f5-49a0-8fa3-4685887748e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1abade-07f5-49a0-8fa3-4685887748e2-istiod-ca-cert\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981330 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-kube-api-access-sfckw\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-kube-api-access-sfckw\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981344 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-certs" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:19.981384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981372 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-data" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:19.981900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981379 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-socket\") pod \"5d1abade-07f5-49a0-8fa3-4685887748e2\" (UID: \"5d1abade-07f5-49a0-8fa3-4685887748e2\") " Nov 03 20:06:19.981900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981570 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-socket" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:19.981900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981701 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1abade-07f5-49a0-8fa3-4685887748e2-istiod-ca-cert" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:19.981900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.981791 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-credential-socket" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:19.982129 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.982099 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.982129 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.982126 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1abade-07f5-49a0-8fa3-4685887748e2-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.982245 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.982142 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.982245 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.982155 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.982245 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.982166 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:19.983679 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.983650 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-token" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:19.983844 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.983823 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-kube-api-access-sfckw" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "kube-api-access-sfckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:19.984049 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.984029 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-envoy" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:19.985394 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:19.985363 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-podinfo" pod "5d1abade-07f5-49a0-8fa3-4685887748e2" (UID: "5d1abade-07f5-49a0-8fa3-4685887748e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:06:20.082961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.082929 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:20.082961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.082960 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfckw\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-kube-api-access-sfckw\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:20.083157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.082970 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:20.083157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.082979 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1abade-07f5-49a0-8fa3-4685887748e2-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:20.159739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.159655 4314 generic.go:356] "Generic (PLEG): container finished" podID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerID="3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8" exitCode=0 Nov 03 20:06:20.159739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.159708 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" Nov 03 20:06:20.159961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.159736 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" event={"ID":"5d1abade-07f5-49a0-8fa3-4685887748e2","Type":"ContainerDied","Data":"3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8"} Nov 03 20:06:20.159961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.159791 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8" event={"ID":"5d1abade-07f5-49a0-8fa3-4685887748e2","Type":"ContainerDied","Data":"0bd4c319cf1b7b04a4bec061983fef5ff46333b9b2d3dcb548f0dedb0311558f"} Nov 03 20:06:20.159961 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.159813 4314 scope.go:122] "RemoveContainer" containerID="3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8" Nov 03 20:06:20.176436 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.176413 4314 scope.go:122] "RemoveContainer" containerID="3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8" Nov 03 20:06:20.176813 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:20.176789 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8\": container with ID starting with 3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8 not found: ID does not exist" containerID="3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8" Nov 03 20:06:20.176875 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.176823 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8"} err="failed to get container status \"3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8\": rpc error: code = NotFound desc = could not find container \"3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8\": container with ID starting with 3466d3c74e46c1a9e50a3b464c4a5d6181c35817ff27d90cd98040bc5cc70fd8 not found: ID does not exist" Nov 03 20:06:20.188608 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.188570 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8"] Nov 03 20:06:20.192621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.192600 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-fcqn8"] Nov 03 20:06:20.429043 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:20.428963 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" path="/var/lib/kubelet/pods/5d1abade-07f5-49a0-8fa3-4685887748e2/volumes" Nov 03 20:06:21.991252 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:21.991212 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:06:26.081335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.081303 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj"] Nov 03 20:06:26.081851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.081511 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" podUID="994e6de3-a05c-4cdd-b5b5-e753e2aeece0" containerName="maas-api" containerID="cri-o://e07f595423e13506e1d08836ae500ce59f6085ac60caaab44aa00b1ff46ded84" gracePeriod=30 Nov 03 20:06:26.236614 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.234358 4314 generic.go:356] "Generic (PLEG): container finished" podID="994e6de3-a05c-4cdd-b5b5-e753e2aeece0" containerID="e07f595423e13506e1d08836ae500ce59f6085ac60caaab44aa00b1ff46ded84" exitCode=0 Nov 03 20:06:26.236614 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.234652 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" event={"ID":"994e6de3-a05c-4cdd-b5b5-e753e2aeece0","Type":"ContainerDied","Data":"e07f595423e13506e1d08836ae500ce59f6085ac60caaab44aa00b1ff46ded84"} Nov 03 20:06:26.482322 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.482290 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 20:06:26.586827 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.586786 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-kube-api-access-l9hwx\" (UniqueName: \"kubernetes.io/projected/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-kube-api-access-l9hwx\") pod \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " Nov 03 20:06:26.587077 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.586898 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-maas-api-tls\") pod \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " Nov 03 20:06:26.587077 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.586980 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-openshift-service-ca\") pod \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " Nov 03 20:06:26.587077 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.587069 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-metrics-certs\") pod \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\" (UID: \"994e6de3-a05c-4cdd-b5b5-e753e2aeece0\") " Nov 03 20:06:26.588357 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.588322 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-openshift-service-ca" pod "994e6de3-a05c-4cdd-b5b5-e753e2aeece0" (UID: "994e6de3-a05c-4cdd-b5b5-e753e2aeece0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:26.589764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.589733 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-metrics-certs" pod "994e6de3-a05c-4cdd-b5b5-e753e2aeece0" (UID: "994e6de3-a05c-4cdd-b5b5-e753e2aeece0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:26.590338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.590305 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-kube-api-access-l9hwx" pod "994e6de3-a05c-4cdd-b5b5-e753e2aeece0" (UID: "994e6de3-a05c-4cdd-b5b5-e753e2aeece0"). InnerVolumeSpecName "kube-api-access-l9hwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:26.590449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.590334 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-maas-api-tls" pod "994e6de3-a05c-4cdd-b5b5-e753e2aeece0" (UID: "994e6de3-a05c-4cdd-b5b5-e753e2aeece0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:26.689084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.689040 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:26.689084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.689070 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hwx\" (UniqueName: \"kubernetes.io/projected/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-kube-api-access-l9hwx\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:26.689084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.689082 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:26.689084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:26.689091 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/994e6de3-a05c-4cdd-b5b5-e753e2aeece0-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:27.243108 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:27.243074 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" Nov 03 20:06:27.243108 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:27.243088 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj" event={"ID":"994e6de3-a05c-4cdd-b5b5-e753e2aeece0","Type":"ContainerDied","Data":"25a69ccaa0cc0305c7828d2f8d0e53b613b844297779864c13d63d579d271594"} Nov 03 20:06:27.243675 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:27.243180 4314 scope.go:122] "RemoveContainer" containerID="e07f595423e13506e1d08836ae500ce59f6085ac60caaab44aa00b1ff46ded84" Nov 03 20:06:27.269087 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:27.268741 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj"] Nov 03 20:06:27.272418 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:27.272393 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d4127-69c49c8667-7gnzj"] Nov 03 20:06:28.428335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:28.428297 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994e6de3-a05c-4cdd-b5b5-e753e2aeece0" path="/var/lib/kubelet/pods/994e6de3-a05c-4cdd-b5b5-e753e2aeece0/volumes" Nov 03 20:06:33.157721 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:33.157688 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:06:33.163847 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:33.163824 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:06:34.554516 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:34.554481 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b"] Nov 03 20:06:34.554970 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:34.554759 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" containerID="cri-o://bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed" gracePeriod=30 Nov 03 20:06:34.675417 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:34.675377 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9"] Nov 03 20:06:34.675685 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:34.675644 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" containerID="cri-o://cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6" gracePeriod=30 Nov 03 20:06:37.643856 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:37.643824 4314 scope.go:122] "RemoveContainer" containerID="7c03cf39c1419594627414ed248977e9e350fd75f75058083334807c2a5361d2" Nov 03 20:06:39.865010 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.864984 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 20:06:39.958634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958599 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-envoy\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.958836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958683 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-socket\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.958836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958764 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-certs\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.958836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958793 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-kube-api-access-gsm4l\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-kube-api-access-gsm4l\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.958836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958823 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-credential-socket\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.959088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958919 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-token\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-token\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.959088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958961 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-podinfo\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.959088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958988 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da839c7-b3f1-42b7-9fcd-709256732a3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da839c7-b3f1-42b7-9fcd-709256732a3b-istiod-ca-cert\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.959088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.958981 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-socket" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:39.959088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959064 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-data\") pod \"2da839c7-b3f1-42b7-9fcd-709256732a3b\" (UID: \"2da839c7-b3f1-42b7-9fcd-709256732a3b\") " Nov 03 20:06:39.959366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959295 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-credential-socket" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:39.959366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959318 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da839c7-b3f1-42b7-9fcd-709256732a3b-istiod-ca-cert" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:39.959960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959607 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-certs" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:39.959960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959680 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-data" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:39.959960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959868 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da839c7-b3f1-42b7-9fcd-709256732a3b-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.959960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959892 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.959960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959909 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.959960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959922 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.959960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.959935 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.961225 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.961197 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-envoy" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:39.961990 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.961966 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-podinfo" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:06:39.962102 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.961998 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-kube-api-access-gsm4l" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "kube-api-access-gsm4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:39.962165 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:39.962149 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-token" pod "2da839c7-b3f1-42b7-9fcd-709256732a3b" (UID: "2da839c7-b3f1-42b7-9fcd-709256732a3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:40.060772 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.060732 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.060772 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.060766 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.060988 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.060781 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da839c7-b3f1-42b7-9fcd-709256732a3b-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.060988 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.060795 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsm4l\" (UniqueName: \"kubernetes.io/projected/2da839c7-b3f1-42b7-9fcd-709256732a3b-kube-api-access-gsm4l\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.235029 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.235004 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.265690 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-credential-socket\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.265745 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-socket\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266045 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-socket" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266274 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-credential-socket" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266423 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e508fa96-3607-45d5-ad64-e806599da0a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e508fa96-3607-45d5-ad64-e806599da0a7-istio-podinfo\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266467 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-envoy\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266559 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-istio-token\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-istio-token\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266630 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-kube-api-access-xgrg9\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-kube-api-access-xgrg9\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266693 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-data\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266756 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e508fa96-3607-45d5-ad64-e806599da0a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e508fa96-3607-45d5-ad64-e806599da0a7-istiod-ca-cert\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.267834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.266806 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-certs\") pod \"e508fa96-3607-45d5-ad64-e806599da0a7\" (UID: \"e508fa96-3607-45d5-ad64-e806599da0a7\") " Nov 03 20:06:40.270601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.268435 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.270601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.269107 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.270601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.269791 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-data" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:40.270601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.270439 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-certs" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:40.270916 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.270888 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e508fa96-3607-45d5-ad64-e806599da0a7-istiod-ca-cert" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:40.288051 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.288020 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-kube-api-access-xgrg9" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "kube-api-access-xgrg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:40.296829 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.296792 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-istio-token" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:40.297100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.296889 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e508fa96-3607-45d5-ad64-e806599da0a7-istio-podinfo" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:06:40.297884 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.297862 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-envoy" pod "e508fa96-3607-45d5-ad64-e806599da0a7" (UID: "e508fa96-3607-45d5-ad64-e806599da0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:40.340639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.340601 4314 generic.go:356] "Generic (PLEG): container finished" podID="e508fa96-3607-45d5-ad64-e806599da0a7" containerID="bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed" exitCode=0 Nov 03 20:06:40.340823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.340676 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" Nov 03 20:06:40.340823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.340686 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" event={"ID":"e508fa96-3607-45d5-ad64-e806599da0a7","Type":"ContainerDied","Data":"bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed"} Nov 03 20:06:40.340823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.340771 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b" event={"ID":"e508fa96-3607-45d5-ad64-e806599da0a7","Type":"ContainerDied","Data":"5b0120a68a84a3f54723046a64f333772b0b671f6ff3702563b347a89ae48a1b"} Nov 03 20:06:40.340823 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.340794 4314 scope.go:122] "RemoveContainer" containerID="bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed" Nov 03 20:06:40.342568 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.342540 4314 generic.go:356] "Generic (PLEG): container finished" podID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerID="cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6" exitCode=0 Nov 03 20:06:40.342749 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.342620 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" Nov 03 20:06:40.342749 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.342620 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" event={"ID":"2da839c7-b3f1-42b7-9fcd-709256732a3b","Type":"ContainerDied","Data":"cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6"} Nov 03 20:06:40.342749 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.342675 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9" event={"ID":"2da839c7-b3f1-42b7-9fcd-709256732a3b","Type":"ContainerDied","Data":"cfc5f329d57f8ab2b9c302037fbb8e281a203fa35de5b3b0abe2c2042578ec9d"} Nov 03 20:06:40.354394 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.354160 4314 scope.go:122] "RemoveContainer" containerID="536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484" Nov 03 20:06:40.370380 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.370348 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e508fa96-3607-45d5-ad64-e806599da0a7-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.370520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.370383 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.370520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.370397 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.370520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.370411 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrg9\" (UniqueName: \"kubernetes.io/projected/e508fa96-3607-45d5-ad64-e806599da0a7-kube-api-access-xgrg9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.370520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.370427 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.370520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.370439 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e508fa96-3607-45d5-ad64-e806599da0a7-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.370520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.370453 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e508fa96-3607-45d5-ad64-e806599da0a7-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.372186 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.372164 4314 scope.go:122] "RemoveContainer" containerID="bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed" Nov 03 20:06:40.372559 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:40.372535 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed\": container with ID starting with bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed not found: ID does not exist" containerID="bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed" Nov 03 20:06:40.372644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.372569 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed"} err="failed to get container status \"bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed\": rpc error: code = NotFound desc = could not find container \"bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed\": container with ID starting with bc5f8db0d39a1eb2ab757494dc371b78c0b0c7c62b1c1799580d16a26d7404ed not found: ID does not exist" Nov 03 20:06:40.372644 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.372610 4314 scope.go:122] "RemoveContainer" containerID="536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484" Nov 03 20:06:40.372913 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:40.372893 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484\": container with ID starting with 536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484 not found: ID does not exist" containerID="536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484" Nov 03 20:06:40.372990 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.372918 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484"} err="failed to get container status \"536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484\": rpc error: code = NotFound desc = could not find container \"536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484\": container with ID starting with 536c8e234cccf5084cf78c83c531b7e929ebffe1524e5a9caa69eb0a5fcdf484 not found: ID does not exist" Nov 03 20:06:40.372990 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.372935 4314 scope.go:122] "RemoveContainer" containerID="cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6" Nov 03 20:06:40.374197 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.374181 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b"] Nov 03 20:06:40.377001 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.376971 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b9cbea-openshift-default-746c95f64f-l8m2b"] Nov 03 20:06:40.384637 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.384613 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9"] Nov 03 20:06:40.385681 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.385150 4314 scope.go:122] "RemoveContainer" containerID="cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6" Nov 03 20:06:40.385681 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:40.385577 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6\": container with ID starting with cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6 not found: ID does not exist" containerID="cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6" Nov 03 20:06:40.385681 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.385623 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6"} err="failed to get container status \"cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6\": rpc error: code = NotFound desc = could not find container \"cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6\": container with ID starting with cd63e704c209f7e00d152776f2b2f1168f81e0205a96de6baec8e5831fbc30d6 not found: ID does not exist" Nov 03 20:06:40.389350 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.389297 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-dk9s9"] Nov 03 20:06:40.427574 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.427551 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" path="/var/lib/kubelet/pods/2da839c7-b3f1-42b7-9fcd-709256732a3b/volumes" Nov 03 20:06:40.428087 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:40.428071 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" path="/var/lib/kubelet/pods/e508fa96-3607-45d5-ad64-e806599da0a7/volumes" Nov 03 20:06:43.659726 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:43.659689 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:06:51.989922 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:06:51.989838 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:06:59.467409 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.467369 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds"] Nov 03 20:06:59.468031 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.467637 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" podUID="706f41ed-2d34-46e8-8509-0bb2ff2e9f68" containerName="maas-api" containerID="cri-o://0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f" gracePeriod=30 Nov 03 20:06:59.794089 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.794062 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 20:06:59.909092 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.909063 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-kube-api-access-p9p7z\" (UniqueName: \"kubernetes.io/projected/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-kube-api-access-p9p7z\") pod \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " Nov 03 20:06:59.909279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.909150 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-metrics-certs\") pod \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " Nov 03 20:06:59.909279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.909221 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-openshift-service-ca\") pod \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " Nov 03 20:06:59.909410 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.909395 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-maas-api-tls\") pod \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\" (UID: \"706f41ed-2d34-46e8-8509-0bb2ff2e9f68\") " Nov 03 20:06:59.909606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.909561 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-openshift-service-ca" pod "706f41ed-2d34-46e8-8509-0bb2ff2e9f68" (UID: "706f41ed-2d34-46e8-8509-0bb2ff2e9f68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:06:59.909936 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.909908 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:59.911173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.911147 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-kube-api-access-p9p7z" pod "706f41ed-2d34-46e8-8509-0bb2ff2e9f68" (UID: "706f41ed-2d34-46e8-8509-0bb2ff2e9f68"). InnerVolumeSpecName "kube-api-access-p9p7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:59.911361 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.911340 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-maas-api-tls" pod "706f41ed-2d34-46e8-8509-0bb2ff2e9f68" (UID: "706f41ed-2d34-46e8-8509-0bb2ff2e9f68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:06:59.911573 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:06:59.911554 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-metrics-certs" pod "706f41ed-2d34-46e8-8509-0bb2ff2e9f68" (UID: "706f41ed-2d34-46e8-8509-0bb2ff2e9f68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:00.010854 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.010767 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:00.010854 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.010804 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9p7z\" (UniqueName: \"kubernetes.io/projected/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-kube-api-access-p9p7z\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:00.010854 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.010815 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706f41ed-2d34-46e8-8509-0bb2ff2e9f68-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:00.496963 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.496929 4314 generic.go:356] "Generic (PLEG): container finished" podID="706f41ed-2d34-46e8-8509-0bb2ff2e9f68" containerID="0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f" exitCode=0 Nov 03 20:07:00.497343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.496993 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" Nov 03 20:07:00.497343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.497019 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" event={"ID":"706f41ed-2d34-46e8-8509-0bb2ff2e9f68","Type":"ContainerDied","Data":"0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f"} Nov 03 20:07:00.497343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.497073 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds" event={"ID":"706f41ed-2d34-46e8-8509-0bb2ff2e9f68","Type":"ContainerDied","Data":"836f52fa5939fcab66440a3c2a9aa2543d2e739ce0e24205c73a540fcab7f8c7"} Nov 03 20:07:00.497343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.497095 4314 scope.go:122] "RemoveContainer" containerID="0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f" Nov 03 20:07:00.507741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.507722 4314 scope.go:122] "RemoveContainer" containerID="0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f" Nov 03 20:07:00.508055 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:07:00.508026 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f\": container with ID starting with 0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f not found: ID does not exist" containerID="0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f" Nov 03 20:07:00.508159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.508064 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f"} err="failed to get container status \"0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f\": rpc error: code = NotFound desc = could not find container \"0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f\": container with ID starting with 0e47ae8475046863922e1f1eba9ebbe170a11ee8489480495f545e785a3bd58f not found: ID does not exist" Nov 03 20:07:00.518300 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.518271 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds"] Nov 03 20:07:00.521039 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:00.521016 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f4e123-7dc68bcbd6-mxqds"] Nov 03 20:07:02.429370 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:02.429336 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706f41ed-2d34-46e8-8509-0bb2ff2e9f68" path="/var/lib/kubelet/pods/706f41ed-2d34-46e8-8509-0bb2ff2e9f68/volumes" Nov 03 20:07:04.955192 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:04.955152 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:07:07.122659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:07.122623 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m"] Nov 03 20:07:07.123537 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:07.123482 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" containerID="cri-o://21edf33b2d7ff1267a4b24cfac6dcd45d5073e1395328cb19f82b3fbacc02204" gracePeriod=30 Nov 03 20:07:09.980572 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:09.980533 4314 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:07:09.980992 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:09.980621 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:07:12.590213 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.590170 4314 generic.go:356] "Generic (PLEG): container finished" podID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerID="21edf33b2d7ff1267a4b24cfac6dcd45d5073e1395328cb19f82b3fbacc02204" exitCode=0 Nov 03 20:07:12.590704 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.590219 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" event={"ID":"e5706b8d-6931-4fcc-bd8a-06009abcebdc","Type":"ContainerDied","Data":"21edf33b2d7ff1267a4b24cfac6dcd45d5073e1395328cb19f82b3fbacc02204"} Nov 03 20:07:12.829407 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.829378 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 20:07:12.866346 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866262 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-token\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-token\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866346 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866310 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-certs\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866346 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866337 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-kube-api-access-fh9c4\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-kube-api-access-fh9c4\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866401 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istiod-ca-cert\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866462 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-socket\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866515 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-credential-socket\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866542 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-data\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866608 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-envoy\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.866687 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866662 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-podinfo\") pod \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\" (UID: \"e5706b8d-6931-4fcc-bd8a-06009abcebdc\") " Nov 03 20:07:12.867036 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866772 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-socket" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:12.867036 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.866949 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-credential-socket" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:12.867144 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.867033 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-data" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:12.867313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.867294 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.867379 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.867318 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.867379 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.867335 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.868757 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.867661 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-certs" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:12.868757 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.867738 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istiod-ca-cert" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:12.868973 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.868953 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-podinfo" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:07:12.870289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.870261 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-kube-api-access-fh9c4" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "kube-api-access-fh9c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:12.870782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.870760 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-token" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:12.871306 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.871274 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-envoy" pod "e5706b8d-6931-4fcc-bd8a-06009abcebdc" (UID: "e5706b8d-6931-4fcc-bd8a-06009abcebdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:12.968344 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.968314 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.968344 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.968344 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.968544 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.968354 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.968544 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.968363 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5706b8d-6931-4fcc-bd8a-06009abcebdc-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.968544 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.968372 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh9c4\" (UniqueName: \"kubernetes.io/projected/e5706b8d-6931-4fcc-bd8a-06009abcebdc-kube-api-access-fh9c4\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:12.968544 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:12.968382 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5706b8d-6931-4fcc-bd8a-06009abcebdc-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:13.599181 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:13.599147 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" Nov 03 20:07:13.599667 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:13.599153 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m" event={"ID":"e5706b8d-6931-4fcc-bd8a-06009abcebdc","Type":"ContainerDied","Data":"b7ef96289a5973e40135844d7a1fef389657bdad477da740ebc4a68055302ab3"} Nov 03 20:07:13.599667 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:13.599290 4314 scope.go:122] "RemoveContainer" containerID="21edf33b2d7ff1267a4b24cfac6dcd45d5073e1395328cb19f82b3fbacc02204" Nov 03 20:07:13.623963 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:13.623927 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m"] Nov 03 20:07:13.626605 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:13.626568 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-rxg5m"] Nov 03 20:07:14.025412 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:14.025372 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f"] Nov 03 20:07:14.027691 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:14.027663 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec4d538f-qvq8f"] Nov 03 20:07:14.428154 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:14.428069 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b8dc8c-de04-48d7-bfc2-87917a582601" path="/var/lib/kubelet/pods/85b8dc8c-de04-48d7-bfc2-87917a582601/volumes" Nov 03 20:07:14.428593 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:14.428560 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" path="/var/lib/kubelet/pods/e5706b8d-6931-4fcc-bd8a-06009abcebdc/volumes" Nov 03 20:07:21.994325 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:07:21.994281 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:07:23.014199 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:23.014157 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:07:26.025991 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:26.025953 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp"] Nov 03 20:07:26.029492 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:26.029461 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9abd7e0-zj2mp"] Nov 03 20:07:26.428779 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:26.428697 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca102aed-069f-48db-b3ca-05f91cc032d1" path="/var/lib/kubelet/pods/ca102aed-069f-48db-b3ca-05f91cc032d1/volumes" Nov 03 20:07:35.951263 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:35.951225 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:07:37.747886 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:37.747851 4314 scope.go:122] "RemoveContainer" containerID="dfaf4842637b31127ca825859a0275acc2f368270b343ff7fac9850c06e1cc05" Nov 03 20:07:37.759043 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:37.759015 4314 scope.go:122] "RemoveContainer" containerID="e1fd576617cff36784c3c79716732a31a50f448d2f80f77e5f7f726048a91e39" Nov 03 20:07:40.085944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:40.085904 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62"] Nov 03 20:07:40.098775 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:40.098746 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f210049-tqp62"] Nov 03 20:07:40.427943 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:40.427864 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802abf8e-180b-4020-af28-64a5010ef933" path="/var/lib/kubelet/pods/802abf8e-180b-4020-af28-64a5010ef933/volumes" Nov 03 20:07:45.710479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:45.710439 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c"] Nov 03 20:07:45.711252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:45.711208 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" podUID="1eabd202-7812-434f-b7d7-9af6b1532f42" containerName="maas-api" containerID="cri-o://8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77" gracePeriod=30 Nov 03 20:07:46.008209 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.008166 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:07:46.092332 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.092292 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-metrics-certs\") pod \"1eabd202-7812-434f-b7d7-9af6b1532f42\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " Nov 03 20:07:46.092564 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.092396 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eabd202-7812-434f-b7d7-9af6b1532f42-kube-api-access-dgrd9\" (UniqueName: \"kubernetes.io/projected/1eabd202-7812-434f-b7d7-9af6b1532f42-kube-api-access-dgrd9\") pod \"1eabd202-7812-434f-b7d7-9af6b1532f42\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " Nov 03 20:07:46.092564 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.092449 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eabd202-7812-434f-b7d7-9af6b1532f42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eabd202-7812-434f-b7d7-9af6b1532f42-openshift-service-ca\") pod \"1eabd202-7812-434f-b7d7-9af6b1532f42\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " Nov 03 20:07:46.092735 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.092659 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls\") pod \"1eabd202-7812-434f-b7d7-9af6b1532f42\" (UID: \"1eabd202-7812-434f-b7d7-9af6b1532f42\") " Nov 03 20:07:46.093353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.093326 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eabd202-7812-434f-b7d7-9af6b1532f42-openshift-service-ca" pod "1eabd202-7812-434f-b7d7-9af6b1532f42" (UID: "1eabd202-7812-434f-b7d7-9af6b1532f42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:46.095058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.095029 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-metrics-certs" pod "1eabd202-7812-434f-b7d7-9af6b1532f42" (UID: "1eabd202-7812-434f-b7d7-9af6b1532f42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:46.095716 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.095677 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls" pod "1eabd202-7812-434f-b7d7-9af6b1532f42" (UID: "1eabd202-7812-434f-b7d7-9af6b1532f42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:46.095716 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.095698 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eabd202-7812-434f-b7d7-9af6b1532f42-kube-api-access-dgrd9" pod "1eabd202-7812-434f-b7d7-9af6b1532f42" (UID: "1eabd202-7812-434f-b7d7-9af6b1532f42"). InnerVolumeSpecName "kube-api-access-dgrd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:46.201975 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.201935 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:46.201975 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.201970 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eabd202-7812-434f-b7d7-9af6b1532f42-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:46.202204 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.201986 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgrd9\" (UniqueName: \"kubernetes.io/projected/1eabd202-7812-434f-b7d7-9af6b1532f42-kube-api-access-dgrd9\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:46.202204 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.202002 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eabd202-7812-434f-b7d7-9af6b1532f42-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:46.837410 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.837379 4314 generic.go:356] "Generic (PLEG): container finished" podID="1eabd202-7812-434f-b7d7-9af6b1532f42" containerID="8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77" exitCode=0 Nov 03 20:07:46.837868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.837444 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" Nov 03 20:07:46.837868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.837468 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" event={"ID":"1eabd202-7812-434f-b7d7-9af6b1532f42","Type":"ContainerDied","Data":"8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77"} Nov 03 20:07:46.837868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.837532 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c" event={"ID":"1eabd202-7812-434f-b7d7-9af6b1532f42","Type":"ContainerDied","Data":"f068b3f3ec869bbd47e27f9bd285576e80ebb822d9786c1912087ec022d9725b"} Nov 03 20:07:46.837868 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.837558 4314 scope.go:122] "RemoveContainer" containerID="8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77" Nov 03 20:07:46.847497 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.847479 4314 scope.go:122] "RemoveContainer" containerID="8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77" Nov 03 20:07:46.847781 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:07:46.847763 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77\": container with ID starting with 8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77 not found: ID does not exist" containerID="8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77" Nov 03 20:07:46.847847 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.847789 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77"} err="failed to get container status \"8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77\": rpc error: code = NotFound desc = could not find container \"8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77\": container with ID starting with 8ad7b227d37b03803c9dfe2e4899b08a2237d6de16a4ebd2cceaa4511e077e77 not found: ID does not exist" Nov 03 20:07:46.857344 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.857316 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c"] Nov 03 20:07:46.862073 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:46.862047 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3ff278-648f5955cd-g9v8c"] Nov 03 20:07:47.641860 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:47.641821 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q"] Nov 03 20:07:48.427305 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:48.427270 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eabd202-7812-434f-b7d7-9af6b1532f42" path="/var/lib/kubelet/pods/1eabd202-7812-434f-b7d7-9af6b1532f42/volumes" Nov 03 20:07:51.212105 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:51.212068 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl"] Nov 03 20:07:51.251071 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:51.251039 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:07:51.990151 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:07:51.990118 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:07:53.566364 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:53.566325 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp"] Nov 03 20:07:53.566966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:53.566607 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" containerID="cri-o://afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308" gracePeriod=30 Nov 03 20:07:54.189356 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.189320 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t"] Nov 03 20:07:54.190089 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190062 4314 cpu_manager.go:414] "containerMap: removing container" podUID="ad7c86d7-e349-4408-be55-626b637e7db6" containerName="payload-processing" Nov 03 20:07:54.190089 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190084 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7c86d7-e349-4408-be55-626b637e7db6" containerName="payload-processing" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190099 4314 cpu_manager.go:414] "containerMap: removing container" podUID="994e6de3-a05c-4cdd-b5b5-e753e2aeece0" containerName="maas-api" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190107 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994e6de3-a05c-4cdd-b5b5-e753e2aeece0" containerName="maas-api" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190118 4314 cpu_manager.go:414] "containerMap: removing container" podUID="1eabd202-7812-434f-b7d7-9af6b1532f42" containerName="maas-api" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190127 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eabd202-7812-434f-b7d7-9af6b1532f42" containerName="maas-api" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190143 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190150 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190161 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190168 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190185 4314 cpu_manager.go:414] "containerMap: removing container" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190192 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190203 4314 cpu_manager.go:414] "containerMap: removing container" podUID="706f41ed-2d34-46e8-8509-0bb2ff2e9f68" containerName="maas-api" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190210 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706f41ed-2d34-46e8-8509-0bb2ff2e9f68" containerName="maas-api" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190221 4314 cpu_manager.go:414] "containerMap: removing container" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190228 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190249 4314 cpu_manager.go:414] "containerMap: removing container" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190256 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190269 4314 cpu_manager.go:414] "containerMap: removing container" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190276 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190289 4314 cpu_manager.go:414] "containerMap: removing container" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" Nov 03 20:07:54.190294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190295 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190313 4314 cpu_manager.go:414] "containerMap: removing container" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190319 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190333 4314 cpu_manager.go:414] "containerMap: removing container" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190341 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190352 4314 cpu_manager.go:414] "containerMap: removing container" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190359 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190369 4314 cpu_manager.go:414] "containerMap: removing container" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190376 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190396 4314 cpu_manager.go:414] "containerMap: removing container" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190404 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190414 4314 cpu_manager.go:414] "containerMap: removing container" podUID="fdf160b8-8393-4590-95f7-f59769271c0b" containerName="revoke-keys" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190421 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf160b8-8393-4590-95f7-f59769271c0b" containerName="revoke-keys" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190438 4314 cpu_manager.go:414] "containerMap: removing container" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190445 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190457 4314 cpu_manager.go:414] "containerMap: removing container" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerName="payload-pre-processing" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190464 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerName="payload-pre-processing" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190571 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5706b8d-6931-4fcc-bd8a-06009abcebdc" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190596 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190608 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d965a1-421d-4ea9-8d6a-ac95bb9e8ea1" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190617 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190628 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="e508fa96-3607-45d5-ad64-e806599da0a7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190637 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d193b3d-3bf2-4b84-a6cc-7e101e9f4fe5" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190646 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="257d4457-e428-4e46-997d-dbab9754eb5d" containerName="main" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190656 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d337f6-4a62-4fa7-a6ec-25f562474a8d" containerName="main" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190663 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf160b8-8393-4590-95f7-f59769271c0b" containerName="revoke-keys" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190673 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="064de3ea-3b4f-483d-8468-62da6e0a19b7" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190683 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc63876-7c5c-4a54-8227-d78f36e4e4d9" containerName="payload-pre-processing" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190691 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7c86d7-e349-4408-be55-626b637e7db6" containerName="payload-processing" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190702 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="994e6de3-a05c-4cdd-b5b5-e753e2aeece0" containerName="maas-api" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190711 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eabd202-7812-434f-b7d7-9af6b1532f42" containerName="maas-api" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190720 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1abade-07f5-49a0-8fa3-4685887748e2" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190727 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190738 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff0d55c-5ae2-45be-8edb-d770f7ffb20c" containerName="main" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190746 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="706f41ed-2d34-46e8-8509-0bb2ff2e9f68" containerName="maas-api" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190754 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da839c7-b3f1-42b7-9fcd-709256732a3b" containerName="istio-proxy" Nov 03 20:07:54.191411 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.190762 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" Nov 03 20:07:54.198033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.197998 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.200949 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.200927 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t"] Nov 03 20:07:54.202051 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.201904 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:07:54.209468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.203869 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:07:54.301128 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.301097 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.301336 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.301244 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.301336 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.301285 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7zk\" (UniqueName: \"kubernetes.io/projected/65ace724-953d-4d0f-b8f5-f0920a0d43d3-kube-api-access-rr7zk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.402517 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.402475 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.402733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.402612 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.402733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.402645 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7zk\" (UniqueName: \"kubernetes.io/projected/65ace724-953d-4d0f-b8f5-f0920a0d43d3-kube-api-access-rr7zk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.403739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.403694 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.405262 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.405231 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.411170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.411136 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7zk\" (UniqueName: \"kubernetes.io/projected/65ace724-953d-4d0f-b8f5-f0920a0d43d3-kube-api-access-rr7zk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.518976 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.518941 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:54.801349 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.801316 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t"] Nov 03 20:07:54.802539 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:07:54.802507 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ace724_953d_4d0f_b8f5_f0920a0d43d3.slice/crio-108e9c7e529aa96918d27fc23084aa06cb0c392e25c446376fe715315c81b8d3 WatchSource:0}: Error finding container 108e9c7e529aa96918d27fc23084aa06cb0c392e25c446376fe715315c81b8d3: Status 404 returned error can't find the container with id 108e9c7e529aa96918d27fc23084aa06cb0c392e25c446376fe715315c81b8d3 Nov 03 20:07:54.889304 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:54.889274 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" event={"ID":"65ace724-953d-4d0f-b8f5-f0920a0d43d3","Type":"ContainerStarted","Data":"108e9c7e529aa96918d27fc23084aa06cb0c392e25c446376fe715315c81b8d3"} Nov 03 20:07:55.898331 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:55.898295 4314 generic.go:356] "Generic (PLEG): container finished" podID="65ace724-953d-4d0f-b8f5-f0920a0d43d3" containerID="07fcaf6b7bcfc748d9e3246441b6c5938d15cb609834c066fb7345e6d4b57c55" exitCode=0 Nov 03 20:07:55.898756 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:55.898378 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" event={"ID":"65ace724-953d-4d0f-b8f5-f0920a0d43d3","Type":"ContainerDied","Data":"07fcaf6b7bcfc748d9e3246441b6c5938d15cb609834c066fb7345e6d4b57c55"} Nov 03 20:07:57.056600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.056559 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:57.132406 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.132372 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-xks-ca\") pod \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " Nov 03 20:07:57.132623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.132436 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-openshift-ca\") pod \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " Nov 03 20:07:57.132623 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.132530 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ace724-953d-4d0f-b8f5-f0920a0d43d3-kube-api-access-rr7zk\" (UniqueName: \"kubernetes.io/projected/65ace724-953d-4d0f-b8f5-f0920a0d43d3-kube-api-access-rr7zk\") pod \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\" (UID: \"65ace724-953d-4d0f-b8f5-f0920a0d43d3\") " Nov 03 20:07:57.132853 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.132824 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-openshift-ca" pod "65ace724-953d-4d0f-b8f5-f0920a0d43d3" (UID: "65ace724-953d-4d0f-b8f5-f0920a0d43d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:57.134987 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.134953 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-xks-ca" pod "65ace724-953d-4d0f-b8f5-f0920a0d43d3" (UID: "65ace724-953d-4d0f-b8f5-f0920a0d43d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:57.135160 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.135046 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ace724-953d-4d0f-b8f5-f0920a0d43d3-kube-api-access-rr7zk" pod "65ace724-953d-4d0f-b8f5-f0920a0d43d3" (UID: "65ace724-953d-4d0f-b8f5-f0920a0d43d3"). InnerVolumeSpecName "kube-api-access-rr7zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:57.233891 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.233818 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:57.233891 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.233844 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ace724-953d-4d0f-b8f5-f0920a0d43d3-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:57.233891 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.233855 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7zk\" (UniqueName: \"kubernetes.io/projected/65ace724-953d-4d0f-b8f5-f0920a0d43d3-kube-api-access-rr7zk\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:57.915885 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.914100 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" event={"ID":"65ace724-953d-4d0f-b8f5-f0920a0d43d3","Type":"ContainerDied","Data":"108e9c7e529aa96918d27fc23084aa06cb0c392e25c446376fe715315c81b8d3"} Nov 03 20:07:57.915885 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.914213 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="108e9c7e529aa96918d27fc23084aa06cb0c392e25c446376fe715315c81b8d3" Nov 03 20:07:57.915885 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:57.914120 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t" Nov 03 20:07:58.851294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.851240 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:07:58.924060 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.924020 4314 generic.go:356] "Generic (PLEG): container finished" podID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerID="afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308" exitCode=0 Nov 03 20:07:58.924258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.924075 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" event={"ID":"1fedb898-5b21-48a4-bf0d-3f603e9a717b","Type":"ContainerDied","Data":"afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308"} Nov 03 20:07:58.924258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.924085 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" Nov 03 20:07:58.924258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.924128 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp" event={"ID":"1fedb898-5b21-48a4-bf0d-3f603e9a717b","Type":"ContainerDied","Data":"dc1bca63b84ddc68fd27ef273daef34e52359394dc08778ee163bb5bc22ebb3e"} Nov 03 20:07:58.924258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.924161 4314 scope.go:122] "RemoveContainer" containerID="afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308" Nov 03 20:07:58.934652 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.934635 4314 scope.go:122] "RemoveContainer" containerID="7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a" Nov 03 20:07:58.943843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.943829 4314 scope.go:122] "RemoveContainer" containerID="afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308" Nov 03 20:07:58.944103 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:07:58.944079 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308\": container with ID starting with afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308 not found: ID does not exist" containerID="afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308" Nov 03 20:07:58.944152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.944111 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308"} err="failed to get container status \"afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308\": rpc error: code = NotFound desc = could not find container \"afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308\": container with ID starting with afa9fe21dc13bc9cc47459674a9085b96f614084235e1bc8d5b58b9a6ab57308 not found: ID does not exist" Nov 03 20:07:58.944152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.944127 4314 scope.go:122] "RemoveContainer" containerID="7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a" Nov 03 20:07:58.944465 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:07:58.944441 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a\": container with ID starting with 7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a not found: ID does not exist" containerID="7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a" Nov 03 20:07:58.944538 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.944468 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a"} err="failed to get container status \"7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a\": rpc error: code = NotFound desc = could not find container \"7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a\": container with ID starting with 7f5fe851e56858d6383b225ff8398d8ac5036bba2cf7e0a884bb30042ff1dc5a not found: ID does not exist" Nov 03 20:07:58.953447 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953422 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-kube-api-access-vd7vc\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-kube-api-access-vd7vc\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953491 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-token\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-token\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953522 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-credential-socket\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953552 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-socket\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953636 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-certs\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953665 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-podinfo\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953864 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953693 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-data\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953864 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953722 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istiod-ca-cert\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953864 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953767 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-envoy\") pod \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\" (UID: \"1fedb898-5b21-48a4-bf0d-3f603e9a717b\") " Nov 03 20:07:58.953864 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953825 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-credential-socket" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:58.954083 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953980 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-certs" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:58.954083 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.953991 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-data" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:58.954194 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.954075 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-socket" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:58.954296 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.954270 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istiod-ca-cert" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:58.954476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.954456 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:58.954566 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.954480 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:58.954566 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.954492 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:58.954566 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.954504 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:58.954566 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.954514 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:58.956222 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.956198 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-kube-api-access-vd7vc" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "kube-api-access-vd7vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:58.956427 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.956405 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-podinfo" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:07:58.956427 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.956417 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-envoy" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:07:58.956674 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:58.956654 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-token" pod "1fedb898-5b21-48a4-bf0d-3f603e9a717b" (UID: "1fedb898-5b21-48a4-bf0d-3f603e9a717b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:59.055562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:59.055534 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:59.055562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:59.055559 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:59.055752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:59.055571 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd7vc\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-kube-api-access-vd7vc\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:59.055752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:59.055611 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fedb898-5b21-48a4-bf0d-3f603e9a717b-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:59.247679 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:59.247616 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp"] Nov 03 20:07:59.250994 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:59.250967 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3ff278-openshift-default-7477ff4559-ncxvp"] Nov 03 20:07:59.946141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:07:59.946106 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:08:00.436064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:00.436014 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" path="/var/lib/kubelet/pods/1fedb898-5b21-48a4-bf0d-3f603e9a717b/volumes" Nov 03 20:08:02.665855 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.665813 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" podUID="48360d9b-b8aa-4a2e-a61e-9287661e9ee8" containerName="main" containerID="cri-o://f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c" gracePeriod=45 Nov 03 20:08:02.747663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.747632 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" podUID="48360d9b-b8aa-4a2e-a61e-9287661e9ee8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.166:8000/ready\": dial tcp 10.134.0.166:8000: connect: connection refused" Nov 03 20:08:02.931971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.931908 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:08:02.949532 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.949503 4314 generic.go:356] "Generic (PLEG): container finished" podID="48360d9b-b8aa-4a2e-a61e-9287661e9ee8" containerID="f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c" exitCode=0 Nov 03 20:08:02.949688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.949544 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" Nov 03 20:08:02.949688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.949566 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" event={"ID":"48360d9b-b8aa-4a2e-a61e-9287661e9ee8","Type":"ContainerDied","Data":"f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c"} Nov 03 20:08:02.949688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.949637 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q" event={"ID":"48360d9b-b8aa-4a2e-a61e-9287661e9ee8","Type":"ContainerDied","Data":"62c55050f53f7df63f4c1b08e00762cf58c141ee11b5dccc0c7d2d5e498d0f1a"} Nov 03 20:08:02.949688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.949660 4314 scope.go:122] "RemoveContainer" containerID="f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c" Nov 03 20:08:02.968121 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.968024 4314 scope.go:122] "RemoveContainer" containerID="f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c" Nov 03 20:08:02.970021 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:02.969987 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c\": container with ID starting with f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c not found: ID does not exist" containerID="f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c" Nov 03 20:08:02.970141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:02.970029 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c"} err="failed to get container status \"f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c\": rpc error: code = NotFound desc = could not find container \"f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c\": container with ID starting with f3a8b4ba2e3a3572bfbab5bb0730b3d9029a6b96a698f253ea0fe83bbd5d885c not found: ID does not exist" Nov 03 20:08:03.001384 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.001355 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-home\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-home\") pod \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " Nov 03 20:08:03.001534 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.001404 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-dshm\") pod \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " Nov 03 20:08:03.001625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.001566 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-kube-api-access-7bzhp\" (UniqueName: \"kubernetes.io/projected/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-kube-api-access-7bzhp\") pod \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " Nov 03 20:08:03.001625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.001619 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tls-certs\" (UniqueName: \"kubernetes.io/secret/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tls-certs\") pod \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " Nov 03 20:08:03.001760 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.001650 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-model-cache\") pod \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " Nov 03 20:08:03.001760 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.001647 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-home" pod "48360d9b-b8aa-4a2e-a61e-9287661e9ee8" (UID: "48360d9b-b8aa-4a2e-a61e-9287661e9ee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:03.002255 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.002227 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tmp-dir\") pod \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\" (UID: \"48360d9b-b8aa-4a2e-a61e-9287661e9ee8\") " Nov 03 20:08:03.002719 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.002522 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-model-cache" pod "48360d9b-b8aa-4a2e-a61e-9287661e9ee8" (UID: "48360d9b-b8aa-4a2e-a61e-9287661e9ee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:03.003654 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.003603 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:03.003654 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.003630 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:03.004622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.002857 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tmp-dir" pod "48360d9b-b8aa-4a2e-a61e-9287661e9ee8" (UID: "48360d9b-b8aa-4a2e-a61e-9287661e9ee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:03.015601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.015563 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-kube-api-access-7bzhp" pod "48360d9b-b8aa-4a2e-a61e-9287661e9ee8" (UID: "48360d9b-b8aa-4a2e-a61e-9287661e9ee8"). InnerVolumeSpecName "kube-api-access-7bzhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:03.015714 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.015636 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tls-certs" pod "48360d9b-b8aa-4a2e-a61e-9287661e9ee8" (UID: "48360d9b-b8aa-4a2e-a61e-9287661e9ee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:03.017133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.017112 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-dshm" pod "48360d9b-b8aa-4a2e-a61e-9287661e9ee8" (UID: "48360d9b-b8aa-4a2e-a61e-9287661e9ee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:03.104440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.104408 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bzhp\" (UniqueName: \"kubernetes.io/projected/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-kube-api-access-7bzhp\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:03.104440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.104439 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:03.104650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.104450 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:03.104650 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.104459 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48360d9b-b8aa-4a2e-a61e-9287661e9ee8-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:03.274463 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.274432 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q"] Nov 03 20:08:03.276744 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:03.276716 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-801162/auth-test-model-801162-kserve-fbfb568c7-nd86q"] Nov 03 20:08:04.429591 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:04.429555 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48360d9b-b8aa-4a2e-a61e-9287661e9ee8" path="/var/lib/kubelet/pods/48360d9b-b8aa-4a2e-a61e-9287661e9ee8/volumes" Nov 03 20:08:06.232754 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.232714 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" podUID="4b19b3c1-36e3-454e-90bc-c8ec8de41799" containerName="main" containerID="cri-o://b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba" gracePeriod=45 Nov 03 20:08:06.498663 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.498638 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:08:06.542063 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542027 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tls-certs\") pod \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " Nov 03 20:08:06.542233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542119 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-home\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-home\") pod \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " Nov 03 20:08:06.542233 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542177 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b19b3c1-36e3-454e-90bc-c8ec8de41799-kube-api-access-2f5xx\" (UniqueName: \"kubernetes.io/projected/4b19b3c1-36e3-454e-90bc-c8ec8de41799-kube-api-access-2f5xx\") pod \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " Nov 03 20:08:06.542366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542236 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-dshm\") pod \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " Nov 03 20:08:06.542366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542289 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tmp-dir\") pod \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " Nov 03 20:08:06.542366 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542333 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-model-cache\") pod \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\" (UID: \"4b19b3c1-36e3-454e-90bc-c8ec8de41799\") " Nov 03 20:08:06.542733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542708 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-home" pod "4b19b3c1-36e3-454e-90bc-c8ec8de41799" (UID: "4b19b3c1-36e3-454e-90bc-c8ec8de41799"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:06.542982 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542904 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-model-cache" pod "4b19b3c1-36e3-454e-90bc-c8ec8de41799" (UID: "4b19b3c1-36e3-454e-90bc-c8ec8de41799"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:06.543096 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.542918 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tmp-dir" pod "4b19b3c1-36e3-454e-90bc-c8ec8de41799" (UID: "4b19b3c1-36e3-454e-90bc-c8ec8de41799"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:06.544911 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.544872 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-dshm" pod "4b19b3c1-36e3-454e-90bc-c8ec8de41799" (UID: "4b19b3c1-36e3-454e-90bc-c8ec8de41799"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:06.545421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.545120 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b19b3c1-36e3-454e-90bc-c8ec8de41799-kube-api-access-2f5xx" pod "4b19b3c1-36e3-454e-90bc-c8ec8de41799" (UID: "4b19b3c1-36e3-454e-90bc-c8ec8de41799"). InnerVolumeSpecName "kube-api-access-2f5xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:06.545421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.545139 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tls-certs" pod "4b19b3c1-36e3-454e-90bc-c8ec8de41799" (UID: "4b19b3c1-36e3-454e-90bc-c8ec8de41799"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:06.643638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.643607 4314 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tls-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:06.643638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.643634 4314 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-home\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:06.643638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.643644 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5xx\" (UniqueName: \"kubernetes.io/projected/4b19b3c1-36e3-454e-90bc-c8ec8de41799-kube-api-access-2f5xx\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:06.643857 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.643654 4314 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-dshm\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:06.643857 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.643664 4314 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-tmp-dir\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:06.643857 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.643672 4314 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b19b3c1-36e3-454e-90bc-c8ec8de41799-model-cache\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:06.986849 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.986755 4314 generic.go:356] "Generic (PLEG): container finished" podID="4b19b3c1-36e3-454e-90bc-c8ec8de41799" containerID="b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba" exitCode=0 Nov 03 20:08:06.987133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.986869 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" Nov 03 20:08:06.987133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.986916 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" event={"ID":"4b19b3c1-36e3-454e-90bc-c8ec8de41799","Type":"ContainerDied","Data":"b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba"} Nov 03 20:08:06.987133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.986958 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl" event={"ID":"4b19b3c1-36e3-454e-90bc-c8ec8de41799","Type":"ContainerDied","Data":"eea2d16ba62f879caafaac51046d37c0d4a1cb31af43baf339356d41082f3e09"} Nov 03 20:08:06.987133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:06.986982 4314 scope.go:122] "RemoveContainer" containerID="b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba" Nov 03 20:08:07.000403 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:07.000379 4314 scope.go:122] "RemoveContainer" containerID="b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba" Nov 03 20:08:07.000836 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:07.000807 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba\": container with ID starting with b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba not found: ID does not exist" containerID="b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba" Nov 03 20:08:07.001004 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:07.000965 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba"} err="failed to get container status \"b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba\": rpc error: code = NotFound desc = could not find container \"b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba\": container with ID starting with b7e0091544010613da7a9647b228be169b0b3b48022f6b126162bd909847a8ba not found: ID does not exist" Nov 03 20:08:07.015633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:07.015577 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl"] Nov 03 20:08:07.017939 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:07.017917 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-b30676/auth-test-model-b30676-kserve-848c9d8b65-hsqzl"] Nov 03 20:08:07.503875 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:07.501019 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t"] Nov 03 20:08:07.504539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:07.504516 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dce558-6cp2t"] Nov 03 20:08:08.427220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:08.427187 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b19b3c1-36e3-454e-90bc-c8ec8de41799" path="/var/lib/kubelet/pods/4b19b3c1-36e3-454e-90bc-c8ec8de41799/volumes" Nov 03 20:08:08.427704 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:08.427682 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ace724-953d-4d0f-b8f5-f0920a0d43d3" path="/var/lib/kubelet/pods/65ace724-953d-4d0f-b8f5-f0920a0d43d3/volumes" Nov 03 20:08:08.470238 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:08.470195 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:08:09.748839 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:09.748809 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth"] Nov 03 20:08:09.749299 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:09.749011 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" containerID="cri-o://9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2" gracePeriod=30 Nov 03 20:08:10.204314 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204225 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv"] Nov 03 20:08:10.204734 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204716 4314 cpu_manager.go:414] "containerMap: removing container" podUID="48360d9b-b8aa-4a2e-a61e-9287661e9ee8" containerName="main" Nov 03 20:08:10.204734 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204733 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48360d9b-b8aa-4a2e-a61e-9287661e9ee8" containerName="main" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204756 4314 cpu_manager.go:414] "containerMap: removing container" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204761 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204779 4314 cpu_manager.go:414] "containerMap: removing container" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204785 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204793 4314 cpu_manager.go:414] "containerMap: removing container" podUID="4b19b3c1-36e3-454e-90bc-c8ec8de41799" containerName="main" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204798 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b19b3c1-36e3-454e-90bc-c8ec8de41799" containerName="main" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204807 4314 cpu_manager.go:414] "containerMap: removing container" podUID="65ace724-953d-4d0f-b8f5-f0920a0d43d3" containerName="revoke-keys" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204812 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ace724-953d-4d0f-b8f5-f0920a0d43d3" containerName="revoke-keys" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204819 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204823 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb09d7b-edcd-4f5b-a297-b700f4b32b59" containerName="istio-proxy" Nov 03 20:08:10.204888 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204894 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" Nov 03 20:08:10.205455 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204906 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fedb898-5b21-48a4-bf0d-3f603e9a717b" containerName="istio-proxy" Nov 03 20:08:10.205455 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204913 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b19b3c1-36e3-454e-90bc-c8ec8de41799" containerName="main" Nov 03 20:08:10.205455 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204920 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ace724-953d-4d0f-b8f5-f0920a0d43d3" containerName="revoke-keys" Nov 03 20:08:10.205455 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.204928 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="48360d9b-b8aa-4a2e-a61e-9287661e9ee8" containerName="main" Nov 03 20:08:10.209641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.209620 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.213655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.213626 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:10.213655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.213638 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:08:10.214189 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.214166 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv"] Nov 03 20:08:10.288516 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.288481 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmvv\" (UniqueName: \"kubernetes.io/projected/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-kube-api-access-qrmvv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.288712 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.288553 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.288712 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.288676 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.390289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.390251 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmvv\" (UniqueName: \"kubernetes.io/projected/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-kube-api-access-qrmvv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.390470 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.390320 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.390470 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.390412 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.391199 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.391167 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.393476 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.393450 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.398087 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.398068 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmvv\" (UniqueName: \"kubernetes.io/projected/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-kube-api-access-qrmvv\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.526287 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.526255 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:10.808684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:10.808656 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv"] Nov 03 20:08:10.811183 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:08:10.811156 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f70af2_0d50_4171_aa3c_62f0d9dc3c93.slice/crio-b3d2604d725e1ad220966ebe39027f18828c4075a59b5644a1d16e1a4bd99a85 WatchSource:0}: Error finding container b3d2604d725e1ad220966ebe39027f18828c4075a59b5644a1d16e1a4bd99a85: Status 404 returned error can't find the container with id b3d2604d725e1ad220966ebe39027f18828c4075a59b5644a1d16e1a4bd99a85 Nov 03 20:08:11.019051 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:11.019011 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" event={"ID":"34f70af2-0d50-4171-aa3c-62f0d9dc3c93","Type":"ContainerStarted","Data":"b3d2604d725e1ad220966ebe39027f18828c4075a59b5644a1d16e1a4bd99a85"} Nov 03 20:08:12.028315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:12.028227 4314 generic.go:356] "Generic (PLEG): container finished" podID="34f70af2-0d50-4171-aa3c-62f0d9dc3c93" containerID="bae0c4d131c74e2745d8e4bcd82db4208f5a47041bf29d1bd9dea4dcb657bfe7" exitCode=0 Nov 03 20:08:12.028315 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:12.028286 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" event={"ID":"34f70af2-0d50-4171-aa3c-62f0d9dc3c93","Type":"ContainerDied","Data":"bae0c4d131c74e2745d8e4bcd82db4208f5a47041bf29d1bd9dea4dcb657bfe7"} Nov 03 20:08:13.189702 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.189677 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:13.326815 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.326729 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-openshift-ca\") pod \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " Nov 03 20:08:13.326815 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.326794 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-xks-ca\") pod \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " Nov 03 20:08:13.327055 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.326838 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-kube-api-access-qrmvv\" (UniqueName: \"kubernetes.io/projected/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-kube-api-access-qrmvv\") pod \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\" (UID: \"34f70af2-0d50-4171-aa3c-62f0d9dc3c93\") " Nov 03 20:08:13.327172 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.327143 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-openshift-ca" pod "34f70af2-0d50-4171-aa3c-62f0d9dc3c93" (UID: "34f70af2-0d50-4171-aa3c-62f0d9dc3c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:13.327412 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.327389 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:13.329335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.329171 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-kube-api-access-qrmvv" pod "34f70af2-0d50-4171-aa3c-62f0d9dc3c93" (UID: "34f70af2-0d50-4171-aa3c-62f0d9dc3c93"). InnerVolumeSpecName "kube-api-access-qrmvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:13.329736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.329712 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-xks-ca" pod "34f70af2-0d50-4171-aa3c-62f0d9dc3c93" (UID: "34f70af2-0d50-4171-aa3c-62f0d9dc3c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:13.428761 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.428724 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrmvv\" (UniqueName: \"kubernetes.io/projected/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-kube-api-access-qrmvv\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:13.428761 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:13.428759 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f70af2-0d50-4171-aa3c-62f0d9dc3c93-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:14.042733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.042690 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" Nov 03 20:08:14.042733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.042724 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv" event={"ID":"34f70af2-0d50-4171-aa3c-62f0d9dc3c93","Type":"ContainerDied","Data":"b3d2604d725e1ad220966ebe39027f18828c4075a59b5644a1d16e1a4bd99a85"} Nov 03 20:08:14.043011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.042795 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d2604d725e1ad220966ebe39027f18828c4075a59b5644a1d16e1a4bd99a85" Nov 03 20:08:14.363828 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.363737 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk"] Nov 03 20:08:14.364435 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.364416 4314 cpu_manager.go:414] "containerMap: removing container" podUID="34f70af2-0d50-4171-aa3c-62f0d9dc3c93" containerName="revoke-keys" Nov 03 20:08:14.364536 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.364437 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f70af2-0d50-4171-aa3c-62f0d9dc3c93" containerName="revoke-keys" Nov 03 20:08:14.364640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.364616 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f70af2-0d50-4171-aa3c-62f0d9dc3c93" containerName="revoke-keys" Nov 03 20:08:14.368509 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.368482 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.376018 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.375958 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-b30676-openshift-default-dockercfg-qk989\"" Nov 03 20:08:14.378629 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.378608 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk"] Nov 03 20:08:14.439052 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439013 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-token\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439104 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7792ade3-4817-45e5-973a-01ebe916d4f2-istiod-ca-cert\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439135 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-certs\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439208 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-envoy\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439383 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439304 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-podinfo\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439383 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439353 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-data\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439500 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439487 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-credential-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439550 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439509 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.439550 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.439533 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwj5\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-kube-api-access-pxwj5\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561096 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561058 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-token\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561146 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7792ade3-4817-45e5-973a-01ebe916d4f2-istiod-ca-cert\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561174 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-certs\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561203 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-envoy\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561232 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-podinfo\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561273 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-data\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561387 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-credential-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561415 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.561617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.561455 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwj5\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-kube-api-access-pxwj5\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.562080 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.562030 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-data\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.562166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.562076 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-credential-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.562166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.562125 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7792ade3-4817-45e5-973a-01ebe916d4f2-istiod-ca-cert\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.562166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.562141 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-socket\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.562480 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.562459 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-certs\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.564610 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.564573 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-podinfo\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.571313 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.571273 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-token\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.571779 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.571674 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwj5\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-kube-api-access-pxwj5\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.578951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.578916 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-envoy\") pod \"e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.687481 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.687393 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:14.896734 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.894710 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t"] Nov 03 20:08:14.896734 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.894972 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" containerID="cri-o://3e3634a7c2862d5f4c51217ebdc94af55fc39dbed3d4845402abe9407bcd1f6c" gracePeriod=30 Nov 03 20:08:14.982899 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.982273 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk"] Nov 03 20:08:15.005847 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.992550 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:15.005847 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:14.992653 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:15.005847 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.005683 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:08:15.044781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.044754 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:08:15.050936 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.050903 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" event={"ID":"7792ade3-4817-45e5-973a-01ebe916d4f2","Type":"ContainerStarted","Data":"96ca03cae22a27d1d86616ed5e314558766834d717530d1416b43662a7a8cd2e"} Nov 03 20:08:15.052688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.052655 4314 generic.go:356] "Generic (PLEG): container finished" podID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerID="9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2" exitCode=0 Nov 03 20:08:15.052765 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.052698 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" Nov 03 20:08:15.052765 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.052708 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" event={"ID":"6612b77c-c2f2-4eb4-a203-2ab47761d23b","Type":"ContainerDied","Data":"9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2"} Nov 03 20:08:15.052842 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.052762 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth" event={"ID":"6612b77c-c2f2-4eb4-a203-2ab47761d23b","Type":"ContainerDied","Data":"8d5eff92c0330e08fc3401710c6829fcfcbde14ce6906713d8b405a478d8e906"} Nov 03 20:08:15.052842 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.052792 4314 scope.go:122] "RemoveContainer" containerID="9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2" Nov 03 20:08:15.082792 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.078878 4314 scope.go:122] "RemoveContainer" containerID="24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937" Nov 03 20:08:15.101457 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.101432 4314 scope.go:122] "RemoveContainer" containerID="9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2" Nov 03 20:08:15.101776 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:15.101748 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2\": container with ID starting with 9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2 not found: ID does not exist" containerID="9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2" Nov 03 20:08:15.101832 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.101788 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2"} err="failed to get container status \"9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2\": rpc error: code = NotFound desc = could not find container \"9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2\": container with ID starting with 9070329a0c6f8d3074b13c21b70f86b0e762694a45f94e3a275d253bf318b2a2 not found: ID does not exist" Nov 03 20:08:15.101832 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.101808 4314 scope.go:122] "RemoveContainer" containerID="24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937" Nov 03 20:08:15.102125 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:15.102104 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937\": container with ID starting with 24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937 not found: ID does not exist" containerID="24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937" Nov 03 20:08:15.102185 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.102128 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937"} err="failed to get container status \"24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937\": rpc error: code = NotFound desc = could not find container \"24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937\": container with ID starting with 24a2dd9712f1f114e77ac012100576da36c765256ced8259154159acd6ef3937 not found: ID does not exist" Nov 03 20:08:15.168284 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168246 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istiod-ca-cert\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168289 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-podinfo\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168326 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-credential-socket\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168369 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-kube-api-access-7rc2h\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-kube-api-access-7rc2h\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168435 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-socket\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168502 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-certs\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168558 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-token\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-token\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168659 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-envoy\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.168722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168696 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istiod-ca-cert" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:15.168931 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168725 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-data\") pod \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\" (UID: \"6612b77c-c2f2-4eb4-a203-2ab47761d23b\") " Nov 03 20:08:15.169011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.168990 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-certs" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:15.170352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.169279 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.170352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.169304 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.170352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.169297 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-socket" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:15.170352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.169336 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-credential-socket" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:15.170352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.169405 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-data" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:15.171919 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.171867 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-podinfo" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:15.171919 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.171885 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-kube-api-access-7rc2h" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "kube-api-access-7rc2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:15.172170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.171987 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-envoy" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:15.173339 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.173272 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-token" pod "6612b77c-c2f2-4eb4-a203-2ab47761d23b" (UID: "6612b77c-c2f2-4eb4-a203-2ab47761d23b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:15.270894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.270852 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.270894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.270882 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.270894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.270892 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.270894 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.270902 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6612b77c-c2f2-4eb4-a203-2ab47761d23b-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.271280 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.270912 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.271280 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.270921 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rc2h\" (UniqueName: \"kubernetes.io/projected/6612b77c-c2f2-4eb4-a203-2ab47761d23b-kube-api-access-7rc2h\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.271280 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.270932 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6612b77c-c2f2-4eb4-a203-2ab47761d23b-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:15.378418 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.378382 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth"] Nov 03 20:08:15.384621 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.384576 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-801162-openshift-default-8499f8c557-nvjth"] Nov 03 20:08:15.528036 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.527953 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth"] Nov 03 20:08:15.528473 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.528456 4314 cpu_manager.go:414] "containerMap: removing container" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" Nov 03 20:08:15.528551 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.528475 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" Nov 03 20:08:15.528683 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.528668 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" Nov 03 20:08:15.528940 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.528689 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" Nov 03 20:08:15.533068 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.533049 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.538178 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.538149 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth"] Nov 03 20:08:15.546539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.546497 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:15.546539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.546526 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:08:15.693304 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.693267 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7j4\" (UniqueName: \"kubernetes.io/projected/c07379ff-cb90-48c5-a354-a878723eda81-kube-api-access-5t7j4\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.693486 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.693442 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.693715 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.693555 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.794385 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.794283 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.794385 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.794340 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7j4\" (UniqueName: \"kubernetes.io/projected/c07379ff-cb90-48c5-a354-a878723eda81-kube-api-access-5t7j4\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.794628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.794426 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.795159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.795123 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.796807 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.796786 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.806020 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.805995 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7j4\" (UniqueName: \"kubernetes.io/projected/c07379ff-cb90-48c5-a354-a878723eda81-kube-api-access-5t7j4\") pod \"maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:15.849100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:15.849074 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:16.092515 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:16.092420 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" event={"ID":"7792ade3-4817-45e5-973a-01ebe916d4f2","Type":"ContainerStarted","Data":"fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1"} Nov 03 20:08:16.115564 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:16.115506 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" podStartSLOduration=2.11549122 podStartE2EDuration="2.11549122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:16.112345601 +0000 UTC m=+2203.817184954" watchObservedRunningTime="2026-11-03 20:08:16.11549122 +0000 UTC m=+2203.820330560" Nov 03 20:08:16.188459 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:16.188106 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth"] Nov 03 20:08:16.190534 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:08:16.190505 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07379ff_cb90_48c5_a354_a878723eda81.slice/crio-9da6f9a4f91828df22f8022f6f689266b6d0d92d7e14b185064a79120bfa10c6 WatchSource:0}: Error finding container 9da6f9a4f91828df22f8022f6f689266b6d0d92d7e14b185064a79120bfa10c6: Status 404 returned error can't find the container with id 9da6f9a4f91828df22f8022f6f689266b6d0d92d7e14b185064a79120bfa10c6 Nov 03 20:08:16.428289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:16.428259 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" path="/var/lib/kubelet/pods/6612b77c-c2f2-4eb4-a203-2ab47761d23b/volumes" Nov 03 20:08:16.688277 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:16.688187 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:16.690173 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:16.690140 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 20:08:16.690331 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:16.690196 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 20:08:17.101966 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:17.101925 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" event={"ID":"c07379ff-cb90-48c5-a354-a878723eda81","Type":"ContainerStarted","Data":"9fbf6a5fe1e556d475086b46db70fb56726cbbbf1a4a801c36adf503899a68b9"} Nov 03 20:08:17.102147 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:17.101985 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" event={"ID":"c07379ff-cb90-48c5-a354-a878723eda81","Type":"ContainerStarted","Data":"9da6f9a4f91828df22f8022f6f689266b6d0d92d7e14b185064a79120bfa10c6"} Nov 03 20:08:17.121192 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:17.121142 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" podStartSLOduration=2.121129327 podStartE2EDuration="2.121129327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:17.118911348 +0000 UTC m=+2204.823750701" watchObservedRunningTime="2026-11-03 20:08:17.121129327 +0000 UTC m=+2204.825968667" Nov 03 20:08:17.232513 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:17.232478 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:08:17.232706 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:17.232547 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:08:17.687949 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:17.687910 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 20:08:17.688986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:17.688567 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 20:08:18.110825 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:18.110792 4314 generic.go:356] "Generic (PLEG): container finished" podID="c07379ff-cb90-48c5-a354-a878723eda81" containerID="9fbf6a5fe1e556d475086b46db70fb56726cbbbf1a4a801c36adf503899a68b9" exitCode=0 Nov 03 20:08:18.111030 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:18.110872 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" event={"ID":"c07379ff-cb90-48c5-a354-a878723eda81","Type":"ContainerDied","Data":"9fbf6a5fe1e556d475086b46db70fb56726cbbbf1a4a801c36adf503899a68b9"} Nov 03 20:08:18.690965 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:18.690929 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 20:08:18.691540 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:18.691508 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 20:08:19.317661 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.317635 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:19.442763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.442672 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07379ff-cb90-48c5-a354-a878723eda81-kube-api-access-5t7j4\" (UniqueName: \"kubernetes.io/projected/c07379ff-cb90-48c5-a354-a878723eda81-kube-api-access-5t7j4\") pod \"c07379ff-cb90-48c5-a354-a878723eda81\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " Nov 03 20:08:19.442763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.442747 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-openshift-ca\") pod \"c07379ff-cb90-48c5-a354-a878723eda81\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " Nov 03 20:08:19.443035 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.442950 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-xks-ca\") pod \"c07379ff-cb90-48c5-a354-a878723eda81\" (UID: \"c07379ff-cb90-48c5-a354-a878723eda81\") " Nov 03 20:08:19.443130 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.443099 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-openshift-ca" pod "c07379ff-cb90-48c5-a354-a878723eda81" (UID: "c07379ff-cb90-48c5-a354-a878723eda81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:19.443335 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.443315 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.445389 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.445339 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-xks-ca" pod "c07379ff-cb90-48c5-a354-a878723eda81" (UID: "c07379ff-cb90-48c5-a354-a878723eda81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:19.445485 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.445458 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07379ff-cb90-48c5-a354-a878723eda81-kube-api-access-5t7j4" pod "c07379ff-cb90-48c5-a354-a878723eda81" (UID: "c07379ff-cb90-48c5-a354-a878723eda81"). InnerVolumeSpecName "kube-api-access-5t7j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:19.544631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.544574 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07379ff-cb90-48c5-a354-a878723eda81-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.544631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.544629 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t7j4\" (UniqueName: \"kubernetes.io/projected/c07379ff-cb90-48c5-a354-a878723eda81-kube-api-access-5t7j4\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:19.688555 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.688512 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 20:08:19.688755 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:19.688573 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 20:08:20.135363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.127286 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" Nov 03 20:08:20.135363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.127993 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth" event={"ID":"c07379ff-cb90-48c5-a354-a878723eda81","Type":"ContainerDied","Data":"9da6f9a4f91828df22f8022f6f689266b6d0d92d7e14b185064a79120bfa10c6"} Nov 03 20:08:20.135363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.128060 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da6f9a4f91828df22f8022f6f689266b6d0d92d7e14b185064a79120bfa10c6" Nov 03 20:08:20.135363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.130627 4314 generic.go:356] "Generic (PLEG): container finished" podID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerID="3e3634a7c2862d5f4c51217ebdc94af55fc39dbed3d4845402abe9407bcd1f6c" exitCode=0 Nov 03 20:08:20.135363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.130776 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" event={"ID":"85f0f2cb-9364-4969-aaca-da82fe5dd836","Type":"ContainerDied","Data":"3e3634a7c2862d5f4c51217ebdc94af55fc39dbed3d4845402abe9407bcd1f6c"} Nov 03 20:08:20.135363 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.130824 4314 scope.go:122] "RemoveContainer" containerID="f11f1f8f5bfbbdacd77940fdd119b174e4de9db4215641c0862af2a5d7ec4ac2" Nov 03 20:08:20.202864 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.202833 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:08:20.250565 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250530 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-data\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.250739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250606 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-certs\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.250739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250638 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-token\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-token\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.250739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250714 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-podinfo\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.250908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250802 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-credential-socket\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.250908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250848 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-socket\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.250908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250845 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-data" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:20.250908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250885 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-certs" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:20.250908 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250904 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-envoy\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.251170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.250985 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-kube-api-access-lx2vc\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-kube-api-access-lx2vc\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.251170 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.251114 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-credential-socket" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:20.251258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.251169 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-socket" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:20.251429 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.251406 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.251562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.251429 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.251562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.251445 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.251562 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.251456 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.253048 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.253025 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-token" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:20.253141 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.253132 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-kube-api-access-lx2vc" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "kube-api-access-lx2vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:20.253189 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.253137 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-envoy" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:20.253685 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.253667 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-podinfo" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:20.312525 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.312491 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl"] Nov 03 20:08:20.312770 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.312743 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" containerID="cri-o://0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613" gracePeriod=30 Nov 03 20:08:20.351764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.351726 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f0f2cb-9364-4969-aaca-da82fe5dd836-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f0f2cb-9364-4969-aaca-da82fe5dd836-istiod-ca-cert\") pod \"85f0f2cb-9364-4969-aaca-da82fe5dd836\" (UID: \"85f0f2cb-9364-4969-aaca-da82fe5dd836\") " Nov 03 20:08:20.352015 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.351999 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.352088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.352015 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx2vc\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-kube-api-access-lx2vc\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.352088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.352025 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.352088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.352034 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85f0f2cb-9364-4969-aaca-da82fe5dd836-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.352088 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.352045 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f0f2cb-9364-4969-aaca-da82fe5dd836-istiod-ca-cert" pod "85f0f2cb-9364-4969-aaca-da82fe5dd836" (UID: "85f0f2cb-9364-4969-aaca-da82fe5dd836"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:20.452866 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.452833 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85f0f2cb-9364-4969-aaca-da82fe5dd836-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:20.688221 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.688183 4314 patch_prober.go:29] interesting pod/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 20:08:20.688372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:20.688240 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 20:08:21.021453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.021415 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn"] Nov 03 20:08:21.023695 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.023673 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936a5ff2-nqpqn"] Nov 03 20:08:21.139707 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.139675 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" Nov 03 20:08:21.140172 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.139713 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t" event={"ID":"85f0f2cb-9364-4969-aaca-da82fe5dd836","Type":"ContainerDied","Data":"9e2a208f7cb9ddbcbbe76d356e5ee18411670005609197f70567ff39023ebf50"} Nov 03 20:08:21.140172 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.139772 4314 scope.go:122] "RemoveContainer" containerID="3e3634a7c2862d5f4c51217ebdc94af55fc39dbed3d4845402abe9407bcd1f6c" Nov 03 20:08:21.173471 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.173270 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t"] Nov 03 20:08:21.194167 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.194121 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-tgd4t"] Nov 03 20:08:21.311012 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.310913 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv"] Nov 03 20:08:21.314373 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.314341 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-b30676-v48xv"] Nov 03 20:08:21.692921 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.692829 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:21.693571 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.693552 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:21.694236 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:21.694215 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:21.990347 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:21.990262 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:08:22.427900 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:22.427867 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f70af2-0d50-4171-aa3c-62f0d9dc3c93" path="/var/lib/kubelet/pods/34f70af2-0d50-4171-aa3c-62f0d9dc3c93/volumes" Nov 03 20:08:22.428319 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:22.428305 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" path="/var/lib/kubelet/pods/85f0f2cb-9364-4969-aaca-da82fe5dd836/volumes" Nov 03 20:08:22.428716 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:22.428703 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b3b538-8dc4-418d-876b-f64f03e4b21f" path="/var/lib/kubelet/pods/a1b3b538-8dc4-418d-876b-f64f03e4b21f/volumes" Nov 03 20:08:23.154647 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:23.154607 4314 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:08:23.154836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:23.154676 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:08:24.508834 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:24.508805 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:08:25.578943 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.578881 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 20:08:25.622432 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622398 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-envoy\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.622613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622456 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-socket\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.622613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622504 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-certs\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.622613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622537 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-kube-api-access-hmvxv\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-kube-api-access-hmvxv\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.622613 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622602 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-token\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-token\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.622854 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622693 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-credential-socket\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.622854 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622732 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-podinfo\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.622854 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622813 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efee946b-04fd-4e86-b1dd-8190c606fb4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efee946b-04fd-4e86-b1dd-8190c606fb4a-istiod-ca-cert\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.623230 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622855 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-data\") pod \"efee946b-04fd-4e86-b1dd-8190c606fb4a\" (UID: \"efee946b-04fd-4e86-b1dd-8190c606fb4a\") " Nov 03 20:08:25.623230 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.622977 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-socket" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:25.623393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.623258 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-credential-socket" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:25.623393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.623275 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-data" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:25.623393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.623332 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.623609 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.623557 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-certs" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:25.623692 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.623605 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efee946b-04fd-4e86-b1dd-8190c606fb4a-istiod-ca-cert" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:25.625293 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.625258 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-envoy" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:25.625500 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.625472 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-podinfo" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:25.625995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.625969 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-kube-api-access-hmvxv" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "kube-api-access-hmvxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:25.627142 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.627110 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-token" pod "efee946b-04fd-4e86-b1dd-8190c606fb4a" (UID: "efee946b-04fd-4e86-b1dd-8190c606fb4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:25.665871 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.665836 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn"] Nov 03 20:08:25.723977 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.723946 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.724172 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.724155 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.724287 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.724275 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efee946b-04fd-4e86-b1dd-8190c606fb4a-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.724364 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.724353 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.724439 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.724425 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.724508 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.724442 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efee946b-04fd-4e86-b1dd-8190c606fb4a-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.724508 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.724457 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmvxv\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-kube-api-access-hmvxv\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:25.724508 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:25.724473 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efee946b-04fd-4e86-b1dd-8190c606fb4a-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:26.175445 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.175408 4314 generic.go:356] "Generic (PLEG): container finished" podID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerID="0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613" exitCode=0 Nov 03 20:08:26.175684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.175492 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" Nov 03 20:08:26.175684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.175494 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" event={"ID":"efee946b-04fd-4e86-b1dd-8190c606fb4a","Type":"ContainerDied","Data":"0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613"} Nov 03 20:08:26.175684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.175559 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl" event={"ID":"efee946b-04fd-4e86-b1dd-8190c606fb4a","Type":"ContainerDied","Data":"e192a7b6d969a5d2ffd1f847b05269d3f4dce7014a78c3a62e5726846835db77"} Nov 03 20:08:26.175684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.175602 4314 scope.go:122] "RemoveContainer" containerID="0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613" Nov 03 20:08:26.186357 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.186340 4314 scope.go:122] "RemoveContainer" containerID="0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613" Nov 03 20:08:26.186670 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:26.186639 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613\": container with ID starting with 0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613 not found: ID does not exist" containerID="0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613" Nov 03 20:08:26.186748 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.186679 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613"} err="failed to get container status \"0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613\": rpc error: code = NotFound desc = could not find container \"0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613\": container with ID starting with 0ebf8a9ee3d89e52cc0e8010951d5bc246466a26ad18bc90b4f3e34038159613 not found: ID does not exist" Nov 03 20:08:26.199084 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.199058 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl"] Nov 03 20:08:26.204270 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.204247 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-rnwjl"] Nov 03 20:08:26.427808 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.427733 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" path="/var/lib/kubelet/pods/efee946b-04fd-4e86-b1dd-8190c606fb4a/volumes" Nov 03 20:08:26.464165 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464131 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7"] Nov 03 20:08:26.464651 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464633 4314 cpu_manager.go:414] "containerMap: removing container" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" Nov 03 20:08:26.464782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464652 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" Nov 03 20:08:26.464782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464682 4314 cpu_manager.go:414] "containerMap: removing container" podUID="c07379ff-cb90-48c5-a354-a878723eda81" containerName="revoke-keys" Nov 03 20:08:26.464782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464690 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07379ff-cb90-48c5-a354-a878723eda81" containerName="revoke-keys" Nov 03 20:08:26.464782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464720 4314 cpu_manager.go:414] "containerMap: removing container" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" Nov 03 20:08:26.464782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464727 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6612b77c-c2f2-4eb4-a203-2ab47761d23b" containerName="istio-proxy" Nov 03 20:08:26.464782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464743 4314 cpu_manager.go:414] "containerMap: removing container" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" Nov 03 20:08:26.464782 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464750 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" Nov 03 20:08:26.465133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464862 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="efee946b-04fd-4e86-b1dd-8190c606fb4a" containerName="istio-proxy" Nov 03 20:08:26.465133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464875 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" Nov 03 20:08:26.465133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464889 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" Nov 03 20:08:26.465133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.464901 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07379ff-cb90-48c5-a354-a878723eda81" containerName="revoke-keys" Nov 03 20:08:26.469776 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.469746 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.476418 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.476393 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7"] Nov 03 20:08:26.476921 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.476899 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:08:26.477055 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.477025 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:26.532244 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.532218 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k264z\" (UniqueName: \"kubernetes.io/projected/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-kube-api-access-k264z\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.532364 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.532288 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.532426 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.532375 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.632921 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.632888 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.633284 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.632934 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.633284 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.633000 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k264z\" (UniqueName: \"kubernetes.io/projected/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-kube-api-access-k264z\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.633666 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.633643 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.635330 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.635302 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.640592 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.640552 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k264z\" (UniqueName: \"kubernetes.io/projected/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-kube-api-access-k264z\") pod \"maas-api-revoke-sub-419207ba-zm4r7\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:26.797525 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:26.797493 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:27.092018 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:27.091931 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7"] Nov 03 20:08:27.095119 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:08:27.095090 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fd2e47_28ba_41d7_af3c_04ffe1b6c66e.slice/crio-51447df7869b956b93523d3a207a65b9cabe9fc1ff61f7201de6e8a494f16a6e WatchSource:0}: Error finding container 51447df7869b956b93523d3a207a65b9cabe9fc1ff61f7201de6e8a494f16a6e: Status 404 returned error can't find the container with id 51447df7869b956b93523d3a207a65b9cabe9fc1ff61f7201de6e8a494f16a6e Nov 03 20:08:27.183688 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:27.183645 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" event={"ID":"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e","Type":"ContainerStarted","Data":"51447df7869b956b93523d3a207a65b9cabe9fc1ff61f7201de6e8a494f16a6e"} Nov 03 20:08:28.192930 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:28.192896 4314 generic.go:356] "Generic (PLEG): container finished" podID="c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" containerID="90bd2165c79796551c657fb21142e6fa6fc7f055f45056a5ccc39e9a91615daa" exitCode=0 Nov 03 20:08:28.193328 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:28.192987 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" event={"ID":"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e","Type":"ContainerDied","Data":"90bd2165c79796551c657fb21142e6fa6fc7f055f45056a5ccc39e9a91615daa"} Nov 03 20:08:28.486620 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:28.486567 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk"] Nov 03 20:08:28.486857 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:28.486832 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" containerID="cri-o://fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1" gracePeriod=30 Nov 03 20:08:29.351555 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.351527 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:29.364205 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.364179 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-xks-ca\") pod \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " Nov 03 20:08:29.364355 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.364222 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-kube-api-access-k264z\" (UniqueName: \"kubernetes.io/projected/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-kube-api-access-k264z\") pod \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " Nov 03 20:08:29.364355 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.364323 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-openshift-ca\") pod \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\" (UID: \"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e\") " Nov 03 20:08:29.364703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.364675 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-openshift-ca" pod "c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" (UID: "c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:29.368959 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.367866 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-kube-api-access-k264z" pod "c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" (UID: "c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e"). InnerVolumeSpecName "kube-api-access-k264z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:29.368959 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.367889 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-xks-ca" pod "c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" (UID: "c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:29.466202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.466108 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:29.466202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.466139 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:29.466202 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:29.466150 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k264z\" (UniqueName: \"kubernetes.io/projected/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e-kube-api-access-k264z\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:30.207595 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.207552 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" Nov 03 20:08:30.207754 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.207541 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7" event={"ID":"c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e","Type":"ContainerDied","Data":"51447df7869b956b93523d3a207a65b9cabe9fc1ff61f7201de6e8a494f16a6e"} Nov 03 20:08:30.207754 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.207701 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51447df7869b956b93523d3a207a65b9cabe9fc1ff61f7201de6e8a494f16a6e" Nov 03 20:08:30.667166 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.667087 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" podUID="93ccfa05-c339-47bf-a71f-32fe3429410b" containerName="payload-processing" containerID="cri-o://0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be" gracePeriod=25 Nov 03 20:08:30.937033 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.937008 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 20:08:30.988962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.988931 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ccfa05-c339-47bf-a71f-32fe3429410b-kube-api-access-bmthq\" (UniqueName: \"kubernetes.io/projected/93ccfa05-c339-47bf-a71f-32fe3429410b-kube-api-access-bmthq\") pod \"93ccfa05-c339-47bf-a71f-32fe3429410b\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " Nov 03 20:08:30.989121 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.989036 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ccfa05-c339-47bf-a71f-32fe3429410b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ccfa05-c339-47bf-a71f-32fe3429410b-plugins-config-volume\") pod \"93ccfa05-c339-47bf-a71f-32fe3429410b\" (UID: \"93ccfa05-c339-47bf-a71f-32fe3429410b\") " Nov 03 20:08:30.989479 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.989446 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ccfa05-c339-47bf-a71f-32fe3429410b-plugins-config-volume" pod "93ccfa05-c339-47bf-a71f-32fe3429410b" (UID: "93ccfa05-c339-47bf-a71f-32fe3429410b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:30.991087 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:30.991066 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ccfa05-c339-47bf-a71f-32fe3429410b-kube-api-access-bmthq" pod "93ccfa05-c339-47bf-a71f-32fe3429410b" (UID: "93ccfa05-c339-47bf-a71f-32fe3429410b"). InnerVolumeSpecName "kube-api-access-bmthq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:31.090289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.090252 4314 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ccfa05-c339-47bf-a71f-32fe3429410b-plugins-config-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:31.090289 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.090284 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmthq\" (UniqueName: \"kubernetes.io/projected/93ccfa05-c339-47bf-a71f-32fe3429410b-kube-api-access-bmthq\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:31.215297 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.215197 4314 generic.go:356] "Generic (PLEG): container finished" podID="93ccfa05-c339-47bf-a71f-32fe3429410b" containerID="0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be" exitCode=0 Nov 03 20:08:31.215297 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.215258 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" Nov 03 20:08:31.215536 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.215253 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" event={"ID":"93ccfa05-c339-47bf-a71f-32fe3429410b","Type":"ContainerDied","Data":"0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be"} Nov 03 20:08:31.215536 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.215402 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn" event={"ID":"93ccfa05-c339-47bf-a71f-32fe3429410b","Type":"ContainerDied","Data":"8d521fc5e80cbb1d0b964c9577ea9a5180e96dcf7e4d4ef8e46e84f3b6a9e13e"} Nov 03 20:08:31.215536 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.215434 4314 scope.go:122] "RemoveContainer" containerID="0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be" Nov 03 20:08:31.225778 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.225760 4314 scope.go:122] "RemoveContainer" containerID="0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be" Nov 03 20:08:31.226071 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:31.226054 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be\": container with ID starting with 0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be not found: ID does not exist" containerID="0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be" Nov 03 20:08:31.226146 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.226078 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be"} err="failed to get container status \"0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be\": rpc error: code = NotFound desc = could not find container \"0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be\": container with ID starting with 0afdd51c9cfb0d655c5d230656f15a535dfb53892b7ee2bb254bb7d065f3f2be not found: ID does not exist" Nov 03 20:08:31.240362 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.240307 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn"] Nov 03 20:08:31.243276 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:31.243252 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-82dbbf-6b6c57d8d7-lkcnn"] Nov 03 20:08:32.435048 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:32.435008 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ccfa05-c339-47bf-a71f-32fe3429410b" path="/var/lib/kubelet/pods/93ccfa05-c339-47bf-a71f-32fe3429410b/volumes" Nov 03 20:08:33.763195 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.763131 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817296 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-socket\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817356 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-certs\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817402 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-envoy\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817429 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-data\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817540 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-kube-api-access-pxwj5\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-kube-api-access-pxwj5\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817576 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-podinfo\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817620 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-credential-socket\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.817741 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817628 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-socket" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:33.818733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817848 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-data" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:33.818733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.817663 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7792ade3-4817-45e5-973a-01ebe916d4f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7792ade3-4817-45e5-973a-01ebe916d4f2-istiod-ca-cert\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.818733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.818020 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-token\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-token\") pod \"7792ade3-4817-45e5-973a-01ebe916d4f2\" (UID: \"7792ade3-4817-45e5-973a-01ebe916d4f2\") " Nov 03 20:08:33.818733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.818029 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7792ade3-4817-45e5-973a-01ebe916d4f2-istiod-ca-cert" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:33.818733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.818146 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-certs" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:33.818733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.818269 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-credential-socket" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:33.819468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.819382 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.819468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.819408 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7792ade3-4817-45e5-973a-01ebe916d4f2-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.819468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.819422 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.819468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.819434 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.819468 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.819448 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.821490 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.821450 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-token" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:33.822008 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.821970 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-kube-api-access-pxwj5" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "kube-api-access-pxwj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:33.822777 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.822752 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-podinfo" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:33.824140 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.824119 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-envoy" pod "7792ade3-4817-45e5-973a-01ebe916d4f2" (UID: "7792ade3-4817-45e5-973a-01ebe916d4f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:33.921072 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.921038 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.921194 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.921078 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxwj5\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-kube-api-access-pxwj5\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.921194 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.921095 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:33.921194 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:33.921108 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7792ade3-4817-45e5-973a-01ebe916d4f2-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:34.237201 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.237120 4314 generic.go:356] "Generic (PLEG): container finished" podID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerID="fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1" exitCode=0 Nov 03 20:08:34.237201 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.237173 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" Nov 03 20:08:34.237399 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.237205 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" event={"ID":"7792ade3-4817-45e5-973a-01ebe916d4f2","Type":"ContainerDied","Data":"fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1"} Nov 03 20:08:34.237399 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.237266 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk" event={"ID":"7792ade3-4817-45e5-973a-01ebe916d4f2","Type":"ContainerDied","Data":"96ca03cae22a27d1d86616ed5e314558766834d717530d1416b43662a7a8cd2e"} Nov 03 20:08:34.237399 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.237291 4314 scope.go:122] "RemoveContainer" containerID="fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1" Nov 03 20:08:34.247895 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.247875 4314 scope.go:122] "RemoveContainer" containerID="fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1" Nov 03 20:08:34.248142 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:34.248122 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1\": container with ID starting with fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1 not found: ID does not exist" containerID="fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1" Nov 03 20:08:34.248206 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.248155 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1"} err="failed to get container status \"fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1\": rpc error: code = NotFound desc = could not find container \"fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1\": container with ID starting with fceb1eae25fb714f940d116fce90d569376b6ad9e82fee74ce903e5c22a895b1 not found: ID does not exist" Nov 03 20:08:34.260937 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.260915 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk"] Nov 03 20:08:34.264733 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.264712 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-b30676-openshift-default-5d7b4d868c-4kpnk"] Nov 03 20:08:34.427333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:34.427305 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" path="/var/lib/kubelet/pods/7792ade3-4817-45e5-973a-01ebe916d4f2/volumes" Nov 03 20:08:37.808123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:37.808061 4314 scope.go:122] "RemoveContainer" containerID="101e06242ccc75924374ed00749d7cf16498415f960a88dfac51312c01437770" Nov 03 20:08:37.817912 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:37.817879 4314 scope.go:122] "RemoveContainer" containerID="2d5e8fcbd5d347cf1ab599541acb4bd9b3f265f2fd0a8e46d5c27a2e6dcba048" Nov 03 20:08:37.826785 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:37.826766 4314 scope.go:122] "RemoveContainer" containerID="18742f29fe7d976a68ae0eee237ab337c3c2fe72724572a22998341563e4884f" Nov 03 20:08:37.835362 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:37.835344 4314 scope.go:122] "RemoveContainer" containerID="4d8be0e97654dc100bc057cef35828c46a1217a76cb7b72a46b89c0b0a778461" Nov 03 20:08:38.019639 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:38.019606 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk"] Nov 03 20:08:38.024041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:38.024013 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64cb1c6c-xvtjk"] Nov 03 20:08:38.428647 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:38.428610 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b152554a-9b5f-4fae-9174-0662f4eda8c5" path="/var/lib/kubelet/pods/b152554a-9b5f-4fae-9174-0662f4eda8c5/volumes" Nov 03 20:08:39.996787 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.996752 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8"] Nov 03 20:08:39.997370 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997348 4314 cpu_manager.go:414] "containerMap: removing container" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" Nov 03 20:08:39.997464 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997373 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" Nov 03 20:08:39.997464 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997386 4314 cpu_manager.go:414] "containerMap: removing container" podUID="c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" containerName="revoke-keys" Nov 03 20:08:39.997464 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997394 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" containerName="revoke-keys" Nov 03 20:08:39.997464 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997442 4314 cpu_manager.go:414] "containerMap: removing container" podUID="93ccfa05-c339-47bf-a71f-32fe3429410b" containerName="payload-processing" Nov 03 20:08:39.997464 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997450 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ccfa05-c339-47bf-a71f-32fe3429410b" containerName="payload-processing" Nov 03 20:08:39.997464 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997465 4314 cpu_manager.go:414] "containerMap: removing container" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" Nov 03 20:08:39.997878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997472 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f0f2cb-9364-4969-aaca-da82fe5dd836" containerName="istio-proxy" Nov 03 20:08:39.997878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997612 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="7792ade3-4817-45e5-973a-01ebe916d4f2" containerName="istio-proxy" Nov 03 20:08:39.997878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997629 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" containerName="revoke-keys" Nov 03 20:08:39.997878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:39.997646 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ccfa05-c339-47bf-a71f-32fe3429410b" containerName="payload-processing" Nov 03 20:08:40.002698 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.002677 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.007422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.007383 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:08:40.009171 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.009050 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8"] Nov 03 20:08:40.009312 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.009188 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:08:40.084814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.084775 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nz5\" (UniqueName: \"kubernetes.io/projected/9b591cf2-f1f6-4481-b00a-7631c2a57326-kube-api-access-l2nz5\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.085046 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.084953 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.085046 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.085028 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.185878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.185841 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nz5\" (UniqueName: \"kubernetes.io/projected/9b591cf2-f1f6-4481-b00a-7631c2a57326-kube-api-access-l2nz5\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.186087 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.185959 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.186087 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.186019 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.186837 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.186810 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.188323 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.188297 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.193566 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.193541 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nz5\" (UniqueName: \"kubernetes.io/projected/9b591cf2-f1f6-4481-b00a-7631c2a57326-kube-api-access-l2nz5\") pod \"maas-api-revoke-sub-2ba81bf8-h6mh8\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.325140 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.325058 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:40.608866 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.608603 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8"] Nov 03 20:08:40.616292 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:08:40.616257 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b591cf2_f1f6_4481_b00a_7631c2a57326.slice/crio-5302ba14984f132e7c669accea014372aaebd1a4a32713f614087f8e42689556 WatchSource:0}: Error finding container 5302ba14984f132e7c669accea014372aaebd1a4a32713f614087f8e42689556: Status 404 returned error can't find the container with id 5302ba14984f132e7c669accea014372aaebd1a4a32713f614087f8e42689556 Nov 03 20:08:40.841078 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:40.841044 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:08:41.011005 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:41.010965 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth"] Nov 03 20:08:41.012914 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:41.012892 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-82dbbf-znkth"] Nov 03 20:08:41.294913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:41.294875 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" event={"ID":"9b591cf2-f1f6-4481-b00a-7631c2a57326","Type":"ContainerStarted","Data":"1132ea61527d6da31359de3a2564701997dcecc34f5f86f907b9989d4522b46d"} Nov 03 20:08:41.295113 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:41.294939 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" event={"ID":"9b591cf2-f1f6-4481-b00a-7631c2a57326","Type":"ContainerStarted","Data":"5302ba14984f132e7c669accea014372aaebd1a4a32713f614087f8e42689556"} Nov 03 20:08:41.316671 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:41.316605 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" podStartSLOduration=2.316571267 podStartE2EDuration="2.316571267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:41.314362005 +0000 UTC m=+2229.019201359" watchObservedRunningTime="2026-11-03 20:08:41.316571267 +0000 UTC m=+2229.021410609" Nov 03 20:08:42.302955 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:42.302791 4314 generic.go:356] "Generic (PLEG): container finished" podID="9b591cf2-f1f6-4481-b00a-7631c2a57326" containerID="1132ea61527d6da31359de3a2564701997dcecc34f5f86f907b9989d4522b46d" exitCode=0 Nov 03 20:08:42.302955 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:42.302888 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" event={"ID":"9b591cf2-f1f6-4481-b00a-7631c2a57326","Type":"ContainerDied","Data":"1132ea61527d6da31359de3a2564701997dcecc34f5f86f907b9989d4522b46d"} Nov 03 20:08:42.427749 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:42.427717 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07379ff-cb90-48c5-a354-a878723eda81" path="/var/lib/kubelet/pods/c07379ff-cb90-48c5-a354-a878723eda81/volumes" Nov 03 20:08:43.445659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.445636 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:43.521239 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.521204 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-openshift-ca\") pod \"9b591cf2-f1f6-4481-b00a-7631c2a57326\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " Nov 03 20:08:43.521442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.521286 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-xks-ca\") pod \"9b591cf2-f1f6-4481-b00a-7631c2a57326\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " Nov 03 20:08:43.521442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.521390 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b591cf2-f1f6-4481-b00a-7631c2a57326-kube-api-access-l2nz5\" (UniqueName: \"kubernetes.io/projected/9b591cf2-f1f6-4481-b00a-7631c2a57326-kube-api-access-l2nz5\") pod \"9b591cf2-f1f6-4481-b00a-7631c2a57326\" (UID: \"9b591cf2-f1f6-4481-b00a-7631c2a57326\") " Nov 03 20:08:43.521662 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.521638 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-openshift-ca" pod "9b591cf2-f1f6-4481-b00a-7631c2a57326" (UID: "9b591cf2-f1f6-4481-b00a-7631c2a57326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:43.523433 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.523402 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b591cf2-f1f6-4481-b00a-7631c2a57326-kube-api-access-l2nz5" pod "9b591cf2-f1f6-4481-b00a-7631c2a57326" (UID: "9b591cf2-f1f6-4481-b00a-7631c2a57326"). InnerVolumeSpecName "kube-api-access-l2nz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:43.523528 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.523478 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-xks-ca" pod "9b591cf2-f1f6-4481-b00a-7631c2a57326" (UID: "9b591cf2-f1f6-4481-b00a-7631c2a57326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:43.623146 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.623077 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2nz5\" (UniqueName: \"kubernetes.io/projected/9b591cf2-f1f6-4481-b00a-7631c2a57326-kube-api-access-l2nz5\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.623146 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.623104 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:43.623146 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:43.623115 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b591cf2-f1f6-4481-b00a-7631c2a57326-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:44.315527 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:44.315495 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" Nov 03 20:08:44.315706 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:44.315543 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8" event={"ID":"9b591cf2-f1f6-4481-b00a-7631c2a57326","Type":"ContainerDied","Data":"5302ba14984f132e7c669accea014372aaebd1a4a32713f614087f8e42689556"} Nov 03 20:08:44.315706 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:44.315608 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5302ba14984f132e7c669accea014372aaebd1a4a32713f614087f8e42689556" Nov 03 20:08:47.587851 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:47.587816 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6"] Nov 03 20:08:47.588368 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:47.588071 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" containerID="cri-o://cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49" gracePeriod=30 Nov 03 20:08:49.961378 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:49.961343 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:08:51.994386 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:51.994318 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:08:52.838706 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:52.838682 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 20:08:53.019256 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019221 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-token\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.019758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019323 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-certs\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.019758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019367 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-socket\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.019758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019397 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istiod-ca-cert\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.019758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019436 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-data\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.019758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019478 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-envoy\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.019758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019515 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-credential-socket\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.019758 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019599 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-kube-api-access-klj7k\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-kube-api-access-klj7k\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.020062 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.019756 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-socket" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:53.021393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.020319 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-certs" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:53.021393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.020388 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istiod-ca-cert" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:08:53.021393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.020507 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-podinfo\") pod \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\" (UID: \"f553b82b-9c58-4eec-ac79-9915f42e7ae9\") " Nov 03 20:08:53.021393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.021347 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-credential-socket" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.021957 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-token" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024316 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-data" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024445 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-envoy" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024888 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024914 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024930 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024946 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024965 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024980 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.030915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.024993 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f553b82b-9c58-4eec-ac79-9915f42e7ae9-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.037350 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.035732 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-podinfo" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:08:53.038203 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.038174 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-kube-api-access-klj7k" pod "f553b82b-9c58-4eec-ac79-9915f42e7ae9" (UID: "f553b82b-9c58-4eec-ac79-9915f42e7ae9"). InnerVolumeSpecName "kube-api-access-klj7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:53.125641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.125604 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klj7k\" (UniqueName: \"kubernetes.io/projected/f553b82b-9c58-4eec-ac79-9915f42e7ae9-kube-api-access-klj7k\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.125641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.125637 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f553b82b-9c58-4eec-ac79-9915f42e7ae9-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:53.374516 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.374419 4314 generic.go:356] "Generic (PLEG): container finished" podID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerID="cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49" exitCode=0 Nov 03 20:08:53.374516 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.374485 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" Nov 03 20:08:53.374516 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.374494 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" event={"ID":"f553b82b-9c58-4eec-ac79-9915f42e7ae9","Type":"ContainerDied","Data":"cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49"} Nov 03 20:08:53.374826 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.374571 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6" event={"ID":"f553b82b-9c58-4eec-ac79-9915f42e7ae9","Type":"ContainerDied","Data":"7cb206517e62c2173ff6e2ccd67166ce187f472fc7e6c4e6f715ace23aef9a68"} Nov 03 20:08:53.374826 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.374618 4314 scope.go:122] "RemoveContainer" containerID="cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49" Nov 03 20:08:53.385929 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.385911 4314 scope.go:122] "RemoveContainer" containerID="cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49" Nov 03 20:08:53.386211 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:08:53.386193 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49\": container with ID starting with cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49 not found: ID does not exist" containerID="cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49" Nov 03 20:08:53.386279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.386218 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49"} err="failed to get container status \"cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49\": rpc error: code = NotFound desc = could not find container \"cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49\": container with ID starting with cb9b85ece6a80c2bfc73315294e3e5fd0ed6320090702d0378e6ddf242007d49 not found: ID does not exist" Nov 03 20:08:53.402308 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.402281 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6"] Nov 03 20:08:53.406412 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:53.406388 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-xclr6"] Nov 03 20:08:54.426554 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:54.426519 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" path="/var/lib/kubelet/pods/f553b82b-9c58-4eec-ac79-9915f42e7ae9/volumes" Nov 03 20:08:56.653730 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:56.653698 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:08:58.781343 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:58.781302 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k"] Nov 03 20:08:58.781972 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:08:58.781547 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" containerID="cri-o://0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f" gracePeriod=30 Nov 03 20:09:01.256046 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.256010 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c547b4b-8ftf8"] Nov 03 20:09:01.256449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.256211 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-86c547b4b-8ftf8" podUID="98acad18-dee5-4a2b-9d7e-bf019690e3aa" containerName="manager" containerID="cri-o://be40491fce5400918243d8cd5b3e774de0a066fccae5abacc7998bcb3408cb6e" gracePeriod=30 Nov 03 20:09:01.423258 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.423226 4314 generic.go:356] "Generic (PLEG): container finished" podID="98acad18-dee5-4a2b-9d7e-bf019690e3aa" containerID="be40491fce5400918243d8cd5b3e774de0a066fccae5abacc7998bcb3408cb6e" exitCode=0 Nov 03 20:09:01.423438 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.423301 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-8ftf8" event={"ID":"98acad18-dee5-4a2b-9d7e-bf019690e3aa","Type":"ContainerDied","Data":"be40491fce5400918243d8cd5b3e774de0a066fccae5abacc7998bcb3408cb6e"} Nov 03 20:09:01.519442 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.519383 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 20:09:01.704159 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.704126 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-webhook-cert\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-webhook-cert\") pod \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " Nov 03 20:09:01.704334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.704181 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-metrics-certs\") pod \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " Nov 03 20:09:01.704334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.704210 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98acad18-dee5-4a2b-9d7e-bf019690e3aa-kube-api-access-287q8\" (UniqueName: \"kubernetes.io/projected/98acad18-dee5-4a2b-9d7e-bf019690e3aa-kube-api-access-287q8\") pod \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\" (UID: \"98acad18-dee5-4a2b-9d7e-bf019690e3aa\") " Nov 03 20:09:01.706250 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.706221 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-metrics-certs" pod "98acad18-dee5-4a2b-9d7e-bf019690e3aa" (UID: "98acad18-dee5-4a2b-9d7e-bf019690e3aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:01.706419 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.706399 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98acad18-dee5-4a2b-9d7e-bf019690e3aa-kube-api-access-287q8" pod "98acad18-dee5-4a2b-9d7e-bf019690e3aa" (UID: "98acad18-dee5-4a2b-9d7e-bf019690e3aa"). InnerVolumeSpecName "kube-api-access-287q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:01.706419 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.706411 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-webhook-cert" pod "98acad18-dee5-4a2b-9d7e-bf019690e3aa" (UID: "98acad18-dee5-4a2b-9d7e-bf019690e3aa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:01.805081 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.805017 4314 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-webhook-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:01.805081 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.805042 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98acad18-dee5-4a2b-9d7e-bf019690e3aa-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:01.805081 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:01.805054 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-287q8\" (UniqueName: \"kubernetes.io/projected/98acad18-dee5-4a2b-9d7e-bf019690e3aa-kube-api-access-287q8\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:02.430418 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:02.430382 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c547b4b-8ftf8" event={"ID":"98acad18-dee5-4a2b-9d7e-bf019690e3aa","Type":"ContainerDied","Data":"f30aeeca09e68f9900247782d1b1fde4b0cd957021f8e68153e33e57ae0b4f72"} Nov 03 20:09:02.430932 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:02.430440 4314 scope.go:122] "RemoveContainer" containerID="be40491fce5400918243d8cd5b3e774de0a066fccae5abacc7998bcb3408cb6e" Nov 03 20:09:02.430932 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:02.430394 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c547b4b-8ftf8" Nov 03 20:09:02.453873 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:02.453848 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c547b4b-8ftf8"] Nov 03 20:09:02.459049 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:02.459019 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86c547b4b-8ftf8"] Nov 03 20:09:03.191055 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:03.191020 4314 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:09:03.191242 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:03.191106 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:09:04.062750 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.062680 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 20:09:04.128522 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128488 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-token\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-token\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.128686 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128551 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ed5324-5bf5-4535-9fe6-042276f0e767-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ed5324-5bf5-4535-9fe6-042276f0e767-istiod-ca-cert\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.128686 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128654 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-socket\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.128830 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128732 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-certs\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.128830 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128803 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-credential-socket\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.128951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128837 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-podinfo\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.128951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128877 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-envoy\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.129068 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128961 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-kube-api-access-nm4vp\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-kube-api-access-nm4vp\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.129068 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128954 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-socket" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:04.129068 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128991 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-data\") pod \"c4ed5324-5bf5-4535-9fe6-042276f0e767\" (UID: \"c4ed5324-5bf5-4535-9fe6-042276f0e767\") " Nov 03 20:09:04.129396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.129343 4314 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.129396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.128988 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ed5324-5bf5-4535-9fe6-042276f0e767-istiod-ca-cert" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:09:04.129396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.129199 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-credential-socket" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:04.129607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.129410 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-certs" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:04.129607 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.129466 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-data" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:04.131521 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.131493 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-envoy" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:09:04.131649 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.131598 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-token" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:04.131703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.131655 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-podinfo" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:09:04.131763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.131738 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-kube-api-access-nm4vp" pod "c4ed5324-5bf5-4535-9fe6-042276f0e767" (UID: "c4ed5324-5bf5-4535-9fe6-042276f0e767"). InnerVolumeSpecName "kube-api-access-nm4vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:04.230722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230691 4314 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-credential-socket\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.230722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230721 4314 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-podinfo\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.230722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230730 4314 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-envoy\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.230948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230739 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm4vp\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-kube-api-access-nm4vp\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.230948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230749 4314 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-data\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.230948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230757 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ed5324-5bf5-4535-9fe6-042276f0e767-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.230948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230765 4314 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ed5324-5bf5-4535-9fe6-042276f0e767-istiod-ca-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.230948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.230773 4314 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ed5324-5bf5-4535-9fe6-042276f0e767-workload-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:04.428736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.428662 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98acad18-dee5-4a2b-9d7e-bf019690e3aa" path="/var/lib/kubelet/pods/98acad18-dee5-4a2b-9d7e-bf019690e3aa/volumes" Nov 03 20:09:04.451638 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.451612 4314 generic.go:356] "Generic (PLEG): container finished" podID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerID="0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f" exitCode=0 Nov 03 20:09:04.451764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.451665 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" Nov 03 20:09:04.451764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.451697 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" event={"ID":"c4ed5324-5bf5-4535-9fe6-042276f0e767","Type":"ContainerDied","Data":"0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f"} Nov 03 20:09:04.451884 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.451768 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k" event={"ID":"c4ed5324-5bf5-4535-9fe6-042276f0e767","Type":"ContainerDied","Data":"b5c73ce328dad9670cd9abb956f9777543585eea968bd74c59d764cec78dc22a"} Nov 03 20:09:04.451884 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.451812 4314 scope.go:122] "RemoveContainer" containerID="0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f" Nov 03 20:09:04.463047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.463024 4314 scope.go:122] "RemoveContainer" containerID="0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f" Nov 03 20:09:04.463366 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:09:04.463342 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f\": container with ID starting with 0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f not found: ID does not exist" containerID="0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f" Nov 03 20:09:04.463444 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.463378 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f"} err="failed to get container status \"0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f\": rpc error: code = NotFound desc = could not find container \"0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f\": container with ID starting with 0a93efe099a171e58b9aa0783eba16dc520cd3ac8b8c73a2430298ccca8d761f not found: ID does not exist" Nov 03 20:09:04.471637 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.471537 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k"] Nov 03 20:09:04.474995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:04.474971 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-8j22k"] Nov 03 20:09:06.426927 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:06.426890 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" path="/var/lib/kubelet/pods/c4ed5324-5bf5-4535-9fe6-042276f0e767/volumes" Nov 03 20:09:11.012037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.011996 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc"] Nov 03 20:09:11.012803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012776 4314 cpu_manager.go:414] "containerMap: removing container" podUID="9b591cf2-f1f6-4481-b00a-7631c2a57326" containerName="revoke-keys" Nov 03 20:09:11.012882 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012807 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b591cf2-f1f6-4481-b00a-7631c2a57326" containerName="revoke-keys" Nov 03 20:09:11.012882 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012847 4314 cpu_manager.go:414] "containerMap: removing container" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" Nov 03 20:09:11.012882 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012856 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" Nov 03 20:09:11.012882 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012871 4314 cpu_manager.go:414] "containerMap: removing container" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" Nov 03 20:09:11.013440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012882 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" Nov 03 20:09:11.013440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012932 4314 cpu_manager.go:414] "containerMap: removing container" podUID="98acad18-dee5-4a2b-9d7e-bf019690e3aa" containerName="manager" Nov 03 20:09:11.013440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.012939 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98acad18-dee5-4a2b-9d7e-bf019690e3aa" containerName="manager" Nov 03 20:09:11.013440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.013072 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="98acad18-dee5-4a2b-9d7e-bf019690e3aa" containerName="manager" Nov 03 20:09:11.013440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.013092 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b591cf2-f1f6-4481-b00a-7631c2a57326" containerName="revoke-keys" Nov 03 20:09:11.013440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.013106 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ed5324-5bf5-4535-9fe6-042276f0e767" containerName="istio-proxy" Nov 03 20:09:11.013440 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.013114 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="f553b82b-9c58-4eec-ac79-9915f42e7ae9" containerName="istio-proxy" Nov 03 20:09:11.017971 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.017948 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.028798 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.028649 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg"] Nov 03 20:09:11.033155 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.033134 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.034803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.034778 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc"] Nov 03 20:09:11.048164 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.048136 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg"] Nov 03 20:09:11.086559 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086528 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4042181-5019-4886-9677-c736a3a2c48d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.086764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086574 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.086764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086693 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.086764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086741 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.086764 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086759 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4042181-5019-4886-9677-c736a3a2c48d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.086983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086777 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.086983 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086921 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d842\" (UniqueName: \"kubernetes.io/projected/951574bf-7a44-46a7-bb31-aa33bdd6792d-kube-api-access-2d842\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.087123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.086987 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.087123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087039 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hftj\" (UniqueName: \"kubernetes.io/projected/c4042181-5019-4886-9677-c736a3a2c48d-kube-api-access-8hftj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.087123 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087076 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.087309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087122 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.087309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087153 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.087309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087175 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.087309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087196 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.087309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087225 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4042181-5019-4886-9677-c736a3a2c48d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.087309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087260 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.087309 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087288 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951574bf-7a44-46a7-bb31-aa33bdd6792d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.087736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.087312 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.188737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188699 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d842\" (UniqueName: \"kubernetes.io/projected/951574bf-7a44-46a7-bb31-aa33bdd6792d-kube-api-access-2d842\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.188737 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188737 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.188968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188755 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hftj\" (UniqueName: \"kubernetes.io/projected/c4042181-5019-4886-9677-c736a3a2c48d-kube-api-access-8hftj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.188968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188776 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.188968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188814 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.188968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188846 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.188968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188877 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.188968 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188906 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.188994 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4042181-5019-4886-9677-c736a3a2c48d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189023 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189051 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951574bf-7a44-46a7-bb31-aa33bdd6792d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189073 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189119 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4042181-5019-4886-9677-c736a3a2c48d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189167 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189213 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189225 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189242 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189270 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4042181-5019-4886-9677-c736a3a2c48d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189273 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189276 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189163 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.189338 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189295 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.190216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189377 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.190216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189428 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.190216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189691 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.190216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.189767 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.190216 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.190105 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4042181-5019-4886-9677-c736a3a2c48d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.190530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.190425 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951574bf-7a44-46a7-bb31-aa33bdd6792d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.191634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.191607 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.191634 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.191622 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4042181-5019-4886-9677-c736a3a2c48d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.192255 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.192237 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.192368 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.192344 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4042181-5019-4886-9677-c736a3a2c48d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.197078 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.197049 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4042181-5019-4886-9677-c736a3a2c48d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.197685 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.197657 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951574bf-7a44-46a7-bb31-aa33bdd6792d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.197873 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.197856 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hftj\" (UniqueName: \"kubernetes.io/projected/c4042181-5019-4886-9677-c736a3a2c48d-kube-api-access-8hftj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-lwvfc\" (UID: \"c4042181-5019-4886-9677-c736a3a2c48d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.197920 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.197888 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d842\" (UniqueName: \"kubernetes.io/projected/951574bf-7a44-46a7-bb31-aa33bdd6792d-kube-api-access-2d842\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h8wkg\" (UID: \"951574bf-7a44-46a7-bb31-aa33bdd6792d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.336801 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.336715 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:11.354179 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.354156 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:11.625750 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.625719 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc"] Nov 03 20:09:11.627851 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:09:11.627815 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4042181_5019_4886_9677_c736a3a2c48d.slice/crio-e8d398e5c092bfe474d08ca457fe9fed35031ccaf4bff5c8b989429539fc9db7 WatchSource:0}: Error finding container e8d398e5c092bfe474d08ca457fe9fed35031ccaf4bff5c8b989429539fc9db7: Status 404 returned error can't find the container with id e8d398e5c092bfe474d08ca457fe9fed35031ccaf4bff5c8b989429539fc9db7 Nov 03 20:09:11.630193 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.630157 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:09:11.630285 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.630229 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:09:11.630330 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.630284 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:09:11.651960 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.651939 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg"] Nov 03 20:09:11.661797 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:09:11.661774 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951574bf_7a44_46a7_bb31_aa33bdd6792d.slice/crio-4ba0d76f43d4b55a6683744995b545c6b7cdaaf3d6bd547cbda6262d7e6d9d3f WatchSource:0}: Error finding container 4ba0d76f43d4b55a6683744995b545c6b7cdaaf3d6bd547cbda6262d7e6d9d3f: Status 404 returned error can't find the container with id 4ba0d76f43d4b55a6683744995b545c6b7cdaaf3d6bd547cbda6262d7e6d9d3f Nov 03 20:09:11.664299 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.664260 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:09:11.664393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.664345 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:09:11.664463 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:11.664391 4314 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 20:09:12.512739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:12.512482 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" event={"ID":"951574bf-7a44-46a7-bb31-aa33bdd6792d","Type":"ContainerStarted","Data":"12f57543deca7c3c72cce780f105f4408cd2627ea7eb52ed826d70de13015de7"} Nov 03 20:09:12.512739 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:12.512539 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" event={"ID":"951574bf-7a44-46a7-bb31-aa33bdd6792d","Type":"ContainerStarted","Data":"4ba0d76f43d4b55a6683744995b545c6b7cdaaf3d6bd547cbda6262d7e6d9d3f"} Nov 03 20:09:12.514388 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:12.514360 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" event={"ID":"c4042181-5019-4886-9677-c736a3a2c48d","Type":"ContainerStarted","Data":"68ab235805acdf4b09cf5a10ea82c8fe38b4aaca00e49a3f124385ebd8ff27f3"} Nov 03 20:09:12.514511 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:12.514402 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" event={"ID":"c4042181-5019-4886-9677-c736a3a2c48d","Type":"ContainerStarted","Data":"e8d398e5c092bfe474d08ca457fe9fed35031ccaf4bff5c8b989429539fc9db7"} Nov 03 20:09:12.533519 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:12.533464 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podStartSLOduration=1.5334506829999999 podStartE2EDuration="1.533450683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:12.532972505 +0000 UTC m=+2260.237811847" watchObservedRunningTime="2026-11-03 20:09:12.533450683 +0000 UTC m=+2260.238290066" Nov 03 20:09:12.552913 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:12.552859 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podStartSLOduration=2.552846648 podStartE2EDuration="2.552846648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:12.551335187 +0000 UTC m=+2260.256174528" watchObservedRunningTime="2026-11-03 20:09:12.552846648 +0000 UTC m=+2260.257685981" Nov 03 20:09:13.338152 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:13.338042 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:13.339569 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:13.339540 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:09:13.339727 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:13.339601 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:09:13.355155 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:13.355138 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:13.356274 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:13.356255 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 20:09:13.356349 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:13.356292 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 20:09:14.338104 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:14.338061 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:09:14.338530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:14.338119 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:09:14.354683 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:14.354658 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 20:09:14.354746 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:14.354700 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 20:09:15.337460 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:15.337424 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:09:15.337668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:15.337480 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:09:15.355253 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:15.355220 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 20:09:15.355632 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:15.355260 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 20:09:16.337756 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:16.337720 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:09:16.337948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:16.337769 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:09:16.354794 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:16.354765 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 20:09:16.354882 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:16.354812 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 20:09:17.337695 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:17.337656 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:09:17.338110 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:17.337706 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:09:17.355525 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:17.355490 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 20:09:17.355637 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:17.355621 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 20:09:18.337530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:18.337500 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:09:18.337699 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:18.337546 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:09:18.355116 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:18.355088 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Nov 03 20:09:18.355456 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:18.355137 4314 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Nov 03 20:09:19.963392 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:19.963278 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:19.963957 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:19.963848 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:19.965005 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:19.964981 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" Nov 03 20:09:20.075956 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:20.075925 4314 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:20.076395 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:20.076369 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:20.077240 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:20.077221 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" Nov 03 20:09:21.989313 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:09:21.989272 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:09:37.911944 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:37.911909 4314 scope.go:122] "RemoveContainer" containerID="8bd2371e945eadd704f349ba09c86d65def4a3e8436150f4af8e69e7b4c1d476" Nov 03 20:09:37.920815 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:37.920790 4314 scope.go:122] "RemoveContainer" containerID="7632a5dfa87a9f3b1a8e4f420d2b84d6ad21a41c2092203fe155e2ba8b049e4d" Nov 03 20:09:41.023052 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:41.023006 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77"] Nov 03 20:09:41.025251 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:41.025218 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87b927e-q9v77"] Nov 03 20:09:42.430131 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:09:42.430092 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57b1a77-c46d-4c91-a5ac-694c6995525b" path="/var/lib/kubelet/pods/b57b1a77-c46d-4c91-a5ac-694c6995525b/volumes" Nov 03 20:09:51.991622 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:09:51.991515 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:10:12.602451 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:12.602415 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:10:21.988894 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:10:21.988857 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:10:28.016135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:28.016101 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:10:31.595606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:31.588123 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:10:34.019398 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:34.019361 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj"] Nov 03 20:10:34.024307 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:34.024280 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad0f6d1-bxjxj"] Nov 03 20:10:34.426684 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:34.426604 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4007995-7031-440b-ba72-08761b3fd345" path="/var/lib/kubelet/pods/d4007995-7031-440b-ba72-08761b3fd345/volumes" Nov 03 20:10:37.956064 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:37.956031 4314 scope.go:122] "RemoveContainer" containerID="572a94a134fce1f13a408651fa6842092949b9436920ae7bee338894699c7a3b" Nov 03 20:10:37.967628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:37.967606 4314 scope.go:122] "RemoveContainer" containerID="8f10fda60f6eea1b4c87e3adaf6b415a0be6a3fea27a666c3ee9b69be84d5991" Nov 03 20:10:37.977505 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:37.977477 4314 scope.go:122] "RemoveContainer" containerID="6d2b5544be0648d3bfc6f90aec459bdc54aa6a46ef0540d9817f8777501916c4" Nov 03 20:10:37.984948 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:37.984928 4314 scope.go:122] "RemoveContainer" containerID="cfae0006e342d85a1c43bae7b58c5ba2d8845b07110ae5a28f729826b0cd9682" Nov 03 20:10:46.750951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:46.750917 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:10:51.015611 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:51.012714 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:51.015611 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:51.012866 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:51.143856 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:51.143802 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:51.144073 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:51.143892 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:51.990492 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:10:51.990448 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:10:59.443347 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:10:59.443310 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:11:21.989754 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:11:21.989703 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:11:23.809380 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:23.809336 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:11:33.192082 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:33.192051 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:11:33.209117 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:33.209090 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:11:33.950025 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:33.949987 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:11:37.849252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:37.849212 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:11:38.026530 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:38.026486 4314 scope.go:122] "RemoveContainer" containerID="9bbed9d386caff2bf561e83dd01e8fd01cdfa13befa6718520a2d2389a9e4ad9" Nov 03 20:11:38.051486 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:38.051461 4314 scope.go:122] "RemoveContainer" containerID="7b72c334327e5b1573804402fa58ce697e331d8405114613f0faa3301fc0ef9d" Nov 03 20:11:38.079962 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:38.079919 4314 scope.go:122] "RemoveContainer" containerID="9af0dd1a141d5d7fe578d89b477bdedb0c53a160b21cfec91a56ed3bdca2bcf1" Nov 03 20:11:38.119157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:38.119128 4314 scope.go:122] "RemoveContainer" containerID="dc23e2981a709ffe888feefca9c49eae09d17c8f73eec0760aecba36b5fdca1f" Nov 03 20:11:51.989805 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:11:51.989768 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:11:54.708443 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:54.708401 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:11:58.352204 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:11:58.352169 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:12:05.888647 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:05.888608 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng"] Nov 03 20:12:05.889269 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:05.888918 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" podUID="82e5d0aa-673f-4bda-b5b0-a9f769389052" containerName="discovery" containerID="cri-o://edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75" gracePeriod=30 Nov 03 20:12:06.150269 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.150203 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 20:12:06.222154 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222119 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-dns-cert\") pod \"82e5d0aa-673f-4bda-b5b0-a9f769389052\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " Nov 03 20:12:06.222353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222203 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-kubeconfig\") pod \"82e5d0aa-673f-4bda-b5b0-a9f769389052\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " Nov 03 20:12:06.222353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222238 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-cacerts\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-cacerts\") pod \"82e5d0aa-673f-4bda-b5b0-a9f769389052\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " Nov 03 20:12:06.222353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222279 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-token\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-token\") pod \"82e5d0aa-673f-4bda-b5b0-a9f769389052\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " Nov 03 20:12:06.222353 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222323 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-ca-configmap\") pod \"82e5d0aa-673f-4bda-b5b0-a9f769389052\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " Nov 03 20:12:06.222600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222414 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e5d0aa-673f-4bda-b5b0-a9f769389052-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e5d0aa-673f-4bda-b5b0-a9f769389052-local-certs\") pod \"82e5d0aa-673f-4bda-b5b0-a9f769389052\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " Nov 03 20:12:06.222600 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222479 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-kube-api-access-pg6g2\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-kube-api-access-pg6g2\") pod \"82e5d0aa-673f-4bda-b5b0-a9f769389052\" (UID: \"82e5d0aa-673f-4bda-b5b0-a9f769389052\") " Nov 03 20:12:06.222858 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.222822 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-ca-configmap" pod "82e5d0aa-673f-4bda-b5b0-a9f769389052" (UID: "82e5d0aa-673f-4bda-b5b0-a9f769389052"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:06.224659 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.224627 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-token" pod "82e5d0aa-673f-4bda-b5b0-a9f769389052" (UID: "82e5d0aa-673f-4bda-b5b0-a9f769389052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:06.224775 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.224665 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e5d0aa-673f-4bda-b5b0-a9f769389052-local-certs" pod "82e5d0aa-673f-4bda-b5b0-a9f769389052" (UID: "82e5d0aa-673f-4bda-b5b0-a9f769389052"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:06.225058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.225035 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-kubeconfig" pod "82e5d0aa-673f-4bda-b5b0-a9f769389052" (UID: "82e5d0aa-673f-4bda-b5b0-a9f769389052"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:06.225058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.225050 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-dns-cert" pod "82e5d0aa-673f-4bda-b5b0-a9f769389052" (UID: "82e5d0aa-673f-4bda-b5b0-a9f769389052"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:06.225155 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.225078 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-cacerts" pod "82e5d0aa-673f-4bda-b5b0-a9f769389052" (UID: "82e5d0aa-673f-4bda-b5b0-a9f769389052"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:06.225196 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.225161 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-kube-api-access-pg6g2" pod "82e5d0aa-673f-4bda-b5b0-a9f769389052" (UID: "82e5d0aa-673f-4bda-b5b0-a9f769389052"). InnerVolumeSpecName "kube-api-access-pg6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:06.323369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.323324 4314 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-kubeconfig\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:06.323369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.323363 4314 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-cacerts\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:06.323369 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.323376 4314 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-token\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:06.323673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.323389 4314 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-ca-configmap\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:06.323673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.323404 4314 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e5d0aa-673f-4bda-b5b0-a9f769389052-local-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:06.323673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.323415 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg6g2\" (UniqueName: \"kubernetes.io/projected/82e5d0aa-673f-4bda-b5b0-a9f769389052-kube-api-access-pg6g2\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:06.323673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.323427 4314 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82e5d0aa-673f-4bda-b5b0-a9f769389052-istio-csr-dns-cert\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:06.587715 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.587673 4314 generic.go:356] "Generic (PLEG): container finished" podID="82e5d0aa-673f-4bda-b5b0-a9f769389052" containerID="edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75" exitCode=0 Nov 03 20:12:06.587927 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.587731 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" Nov 03 20:12:06.587927 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.587759 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" event={"ID":"82e5d0aa-673f-4bda-b5b0-a9f769389052","Type":"ContainerDied","Data":"edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75"} Nov 03 20:12:06.587927 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.587815 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng" event={"ID":"82e5d0aa-673f-4bda-b5b0-a9f769389052","Type":"ContainerDied","Data":"a43a7b1acf385fb037cdd584fbf8dc3ceab658ba9ae9e17e21b7f2cde8fba235"} Nov 03 20:12:06.587927 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.587835 4314 scope.go:122] "RemoveContainer" containerID="edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75" Nov 03 20:12:06.599330 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.599308 4314 scope.go:122] "RemoveContainer" containerID="edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75" Nov 03 20:12:06.599697 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:12:06.599674 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75\": container with ID starting with edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75 not found: ID does not exist" containerID="edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75" Nov 03 20:12:06.599763 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.599704 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75"} err="failed to get container status \"edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75\": rpc error: code = NotFound desc = could not find container \"edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75\": container with ID starting with edd4950899897e583d07e1ac85f5cb3119dbbb44f68296ef00338e51b5b43e75 not found: ID does not exist" Nov 03 20:12:06.606228 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.606196 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng"] Nov 03 20:12:06.609438 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:06.609411 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9r5ng"] Nov 03 20:12:08.426318 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:08.426283 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e5d0aa-673f-4bda-b5b0-a9f769389052" path="/var/lib/kubelet/pods/82e5d0aa-673f-4bda-b5b0-a9f769389052/volumes" Nov 03 20:12:12.816628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:12.816570 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:12:17.045396 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:17.045344 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:12:21.988736 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:12:21.988700 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:12:31.527090 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:31.527049 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:12:35.077264 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:35.077214 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:12:36.010622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:36.009712 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:12:36.010622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:36.009807 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lwvfc" podUID="c4042181-5019-4886-9677-c736a3a2c48d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:36.085606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:36.084907 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:12:36.085606 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:36.085038 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h8wkg" podUID="951574bf-7a44-46a7-bb31-aa33bdd6792d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:38.244864 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:38.244831 4314 scope.go:122] "RemoveContainer" containerID="38a0f24c24bd9b8cce2b817afe0fa230fe3276185500f4a98d437264e1a48761" Nov 03 20:12:49.257417 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:49.257331 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:12:51.997907 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:12:51.997862 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:12:59.878268 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:12:59.878222 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:13:04.039001 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:04.038961 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:13:21.989717 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:13:21.989678 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:13:25.005803 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:25.005756 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:13:30.027370 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:30.027335 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7"] Nov 03 20:13:30.029429 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:30.029397 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419207ba-zm4r7"] Nov 03 20:13:30.426390 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:30.426306 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e" path="/var/lib/kubelet/pods/c7fd2e47-28ba-41d7-af3c-04ffe1b6c66e/volumes" Nov 03 20:13:43.122041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:43.121995 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:13:44.021730 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:44.021680 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8"] Nov 03 20:13:44.024450 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:44.024418 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba81bf8-h6mh8"] Nov 03 20:13:44.433986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:44.433890 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b591cf2-f1f6-4481-b00a-7631c2a57326" path="/var/lib/kubelet/pods/9b591cf2-f1f6-4481-b00a-7631c2a57326/volumes" Nov 03 20:13:51.989639 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:13:51.989594 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:13:53.322438 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:13:53.322388 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:14:20.532024 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:14:20.531939 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:14:21.992102 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:14:21.992064 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:14:35.851915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:14:35.851874 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:14:38.276817 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:14:38.276781 4314 scope.go:122] "RemoveContainer" containerID="90bd2165c79796551c657fb21142e6fa6fc7f055f45056a5ccc39e9a91615daa" Nov 03 20:14:38.285446 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:14:38.285425 4314 scope.go:122] "RemoveContainer" containerID="bae0c4d131c74e2745d8e4bcd82db4208f5a47041bf29d1bd9dea4dcb657bfe7" Nov 03 20:14:38.293822 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:14:38.293793 4314 scope.go:122] "RemoveContainer" containerID="07fcaf6b7bcfc748d9e3246441b6c5938d15cb609834c066fb7345e6d4b57c55" Nov 03 20:14:38.302458 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:14:38.302439 4314 scope.go:122] "RemoveContainer" containerID="9fbf6a5fe1e556d475086b46db70fb56726cbbbf1a4a801c36adf503899a68b9" Nov 03 20:14:51.386547 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:14:51.386509 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:14:51.990758 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:14:51.990715 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:15:03.349224 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.349183 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc"] Nov 03 20:15:03.349736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.349611 4314 cpu_manager.go:414] "containerMap: removing container" podUID="82e5d0aa-673f-4bda-b5b0-a9f769389052" containerName="discovery" Nov 03 20:15:03.349736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.349623 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e5d0aa-673f-4bda-b5b0-a9f769389052" containerName="discovery" Nov 03 20:15:03.349736 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.349712 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e5d0aa-673f-4bda-b5b0-a9f769389052" containerName="discovery" Nov 03 20:15:03.352781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.352759 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.356766 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.356729 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:03.358201 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.358173 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:15:03.361100 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.360533 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc"] Nov 03 20:15:03.497755 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.497719 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.497967 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.497764 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.497967 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.497909 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v64r\" (UniqueName: \"kubernetes.io/projected/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-kube-api-access-7v64r\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.598694 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.598653 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.598694 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.598697 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.598978 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.598766 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v64r\" (UniqueName: \"kubernetes.io/projected/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-kube-api-access-7v64r\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.599352 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.599299 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.601079 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.601058 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.607489 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.607460 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v64r\" (UniqueName: \"kubernetes.io/projected/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-kube-api-access-7v64r\") pod \"maas-api-revoke-sub-6338e751-n5ljc\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.667449 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.667410 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:03.952439 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:03.952402 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc"] Nov 03 20:15:03.954551 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:15:03.954523 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40991a6_ace3_4956_9dd7_42c1c6f80ff8.slice/crio-d6adc5d57e4b6bb3443c2a2cf63ac975833018ef69dae0f524114da48c3c40b8 WatchSource:0}: Error finding container d6adc5d57e4b6bb3443c2a2cf63ac975833018ef69dae0f524114da48c3c40b8: Status 404 returned error can't find the container with id d6adc5d57e4b6bb3443c2a2cf63ac975833018ef69dae0f524114da48c3c40b8 Nov 03 20:15:04.577601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:04.577499 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" event={"ID":"b40991a6-ace3-4956-9dd7-42c1c6f80ff8","Type":"ContainerStarted","Data":"5bc48cdde80b380f008d5d9cee724812c57fb0444afe00c2b52e2f6bf65aeef0"} Nov 03 20:15:04.577601 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:04.577563 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" event={"ID":"b40991a6-ace3-4956-9dd7-42c1c6f80ff8","Type":"ContainerStarted","Data":"d6adc5d57e4b6bb3443c2a2cf63ac975833018ef69dae0f524114da48c3c40b8"} Nov 03 20:15:04.593752 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:04.593675 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" podStartSLOduration=1.593657219 podStartE2EDuration="1.593657219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:04.593358903 +0000 UTC m=+2612.298198247" watchObservedRunningTime="2026-11-03 20:15:04.593657219 +0000 UTC m=+2612.298496552" Nov 03 20:15:05.584227 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:05.584191 4314 generic.go:356] "Generic (PLEG): container finished" podID="b40991a6-ace3-4956-9dd7-42c1c6f80ff8" containerID="5bc48cdde80b380f008d5d9cee724812c57fb0444afe00c2b52e2f6bf65aeef0" exitCode=0 Nov 03 20:15:05.584668 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:05.584275 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" event={"ID":"b40991a6-ace3-4956-9dd7-42c1c6f80ff8","Type":"ContainerDied","Data":"5bc48cdde80b380f008d5d9cee724812c57fb0444afe00c2b52e2f6bf65aeef0"} Nov 03 20:15:06.731453 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.731421 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:06.833037 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.832999 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-kube-api-access-7v64r\" (UniqueName: \"kubernetes.io/projected/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-kube-api-access-7v64r\") pod \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " Nov 03 20:15:06.833325 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.833175 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-openshift-ca\") pod \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " Nov 03 20:15:06.833325 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.833195 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-xks-ca\") pod \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\" (UID: \"b40991a6-ace3-4956-9dd7-42c1c6f80ff8\") " Nov 03 20:15:06.833655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.833628 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-openshift-ca" pod "b40991a6-ace3-4956-9dd7-42c1c6f80ff8" (UID: "b40991a6-ace3-4956-9dd7-42c1c6f80ff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:06.835199 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.835170 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-kube-api-access-7v64r" pod "b40991a6-ace3-4956-9dd7-42c1c6f80ff8" (UID: "b40991a6-ace3-4956-9dd7-42c1c6f80ff8"). InnerVolumeSpecName "kube-api-access-7v64r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:06.835400 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.835380 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-xks-ca" pod "b40991a6-ace3-4956-9dd7-42c1c6f80ff8" (UID: "b40991a6-ace3-4956-9dd7-42c1c6f80ff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:06.934995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.934957 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:06.934995 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.934994 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:06.935204 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:06.935008 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v64r\" (UniqueName: \"kubernetes.io/projected/b40991a6-ace3-4956-9dd7-42c1c6f80ff8-kube-api-access-7v64r\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:07.595507 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:07.595477 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" Nov 03 20:15:07.595718 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:07.595482 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc" event={"ID":"b40991a6-ace3-4956-9dd7-42c1c6f80ff8","Type":"ContainerDied","Data":"d6adc5d57e4b6bb3443c2a2cf63ac975833018ef69dae0f524114da48c3c40b8"} Nov 03 20:15:07.595718 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:07.595621 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6adc5d57e4b6bb3443c2a2cf63ac975833018ef69dae0f524114da48c3c40b8" Nov 03 20:15:14.363451 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.363412 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8"] Nov 03 20:15:14.363909 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.363885 4314 cpu_manager.go:414] "containerMap: removing container" podUID="b40991a6-ace3-4956-9dd7-42c1c6f80ff8" containerName="revoke-keys" Nov 03 20:15:14.363909 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.363899 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40991a6-ace3-4956-9dd7-42c1c6f80ff8" containerName="revoke-keys" Nov 03 20:15:14.364011 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.363998 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40991a6-ace3-4956-9dd7-42c1c6f80ff8" containerName="revoke-keys" Nov 03 20:15:14.367387 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.367370 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.375813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.373185 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:14.375813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.374402 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\"" Nov 03 20:15:14.378665 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.378636 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8"] Nov 03 20:15:14.404484 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.404443 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.404703 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.404575 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.404781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.404699 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5j75\" (UniqueName: \"kubernetes.io/projected/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-kube-api-access-t5j75\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.505286 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.505234 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.505503 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.505343 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.505503 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.505410 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5j75\" (UniqueName: \"kubernetes.io/projected/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-kube-api-access-t5j75\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.506158 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.506127 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.508150 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.508123 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.513674 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.513649 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5j75\" (UniqueName: \"kubernetes.io/projected/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-kube-api-access-t5j75\") pod \"maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.689830 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.689737 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:14.970183 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:14.970152 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8"] Nov 03 20:15:14.973054 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:15:14.973017 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0a6767_efec_4bee_bbfc_63a8a0f508e8.slice/crio-d745ad41a13031397a4f023575350feedeebcb7ba02313e180e251ff28374be5 WatchSource:0}: Error finding container d745ad41a13031397a4f023575350feedeebcb7ba02313e180e251ff28374be5: Status 404 returned error can't find the container with id d745ad41a13031397a4f023575350feedeebcb7ba02313e180e251ff28374be5 Nov 03 20:15:15.158850 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:15.158812 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:15:15.634722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:15.634625 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" event={"ID":"1e0a6767-efec-4bee-bbfc-63a8a0f508e8","Type":"ContainerStarted","Data":"1c07e01f0f9ae1cb5ae0601d5632a483302cdc86a4a186758659808d8224dd45"} Nov 03 20:15:15.634722 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:15.634692 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" event={"ID":"1e0a6767-efec-4bee-bbfc-63a8a0f508e8","Type":"ContainerStarted","Data":"d745ad41a13031397a4f023575350feedeebcb7ba02313e180e251ff28374be5"} Nov 03 20:15:15.652567 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:15.652506 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" podStartSLOduration=1.652489776 podStartE2EDuration="1.652489776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:15.651072298 +0000 UTC m=+2623.355911639" watchObservedRunningTime="2026-11-03 20:15:15.652489776 +0000 UTC m=+2623.357329115" Nov 03 20:15:16.641493 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:16.641454 4314 generic.go:356] "Generic (PLEG): container finished" podID="1e0a6767-efec-4bee-bbfc-63a8a0f508e8" containerID="1c07e01f0f9ae1cb5ae0601d5632a483302cdc86a4a186758659808d8224dd45" exitCode=0 Nov 03 20:15:16.641979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:16.641535 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" event={"ID":"1e0a6767-efec-4bee-bbfc-63a8a0f508e8","Type":"ContainerDied","Data":"1c07e01f0f9ae1cb5ae0601d5632a483302cdc86a4a186758659808d8224dd45"} Nov 03 20:15:17.790676 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.790647 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:17.838919 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.838881 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-kube-api-access-t5j75\" (UniqueName: \"kubernetes.io/projected/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-kube-api-access-t5j75\") pod \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " Nov 03 20:15:17.839134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.838973 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-openshift-ca\") pod \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " Nov 03 20:15:17.839134 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.839035 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-xks-ca\") pod \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\" (UID: \"1e0a6767-efec-4bee-bbfc-63a8a0f508e8\") " Nov 03 20:15:17.839427 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.839397 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-openshift-ca" pod "1e0a6767-efec-4bee-bbfc-63a8a0f508e8" (UID: "1e0a6767-efec-4bee-bbfc-63a8a0f508e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:17.841157 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.841129 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-kube-api-access-t5j75" pod "1e0a6767-efec-4bee-bbfc-63a8a0f508e8" (UID: "1e0a6767-efec-4bee-bbfc-63a8a0f508e8"). InnerVolumeSpecName "kube-api-access-t5j75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:17.841290 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.841254 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-xks-ca" pod "1e0a6767-efec-4bee-bbfc-63a8a0f508e8" (UID: "1e0a6767-efec-4bee-bbfc-63a8a0f508e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:17.940422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.940337 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5j75\" (UniqueName: \"kubernetes.io/projected/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-kube-api-access-t5j75\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.940422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.940368 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-openshift-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.940422 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:17.940378 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a6767-efec-4bee-bbfc-63a8a0f508e8-maas-api-xks-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.653169 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:18.653139 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" Nov 03 20:15:18.653169 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:18.653157 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8" event={"ID":"1e0a6767-efec-4bee-bbfc-63a8a0f508e8","Type":"ContainerDied","Data":"d745ad41a13031397a4f023575350feedeebcb7ba02313e180e251ff28374be5"} Nov 03 20:15:18.653399 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:18.653206 4314 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d745ad41a13031397a4f023575350feedeebcb7ba02313e180e251ff28374be5" Nov 03 20:15:21.989556 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:15:21.989512 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:15:25.989280 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:25.989187 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8"] Nov 03 20:15:25.993050 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:25.993020 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-030599-h5fr8"] Nov 03 20:15:26.426836 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:26.426794 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0a6767-efec-4bee-bbfc-63a8a0f508e8" path="/var/lib/kubelet/pods/1e0a6767-efec-4bee-bbfc-63a8a0f508e8/volumes" Nov 03 20:15:27.411334 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:27.411292 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:15:38.340878 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:38.340843 4314 scope.go:122] "RemoveContainer" containerID="1132ea61527d6da31359de3a2564701997dcecc34f5f86f907b9989d4522b46d" Nov 03 20:15:49.044798 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.044757 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd"] Nov 03 20:15:49.047520 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.044975 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" podUID="8cd08483-1bd3-46f7-a415-47c5fc688cf8" containerName="manager" containerID="cri-o://90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d" gracePeriod=10 Nov 03 20:15:49.303854 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.303780 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 20:15:49.457967 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.457923 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cd08483-1bd3-46f7-a415-47c5fc688cf8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cd08483-1bd3-46f7-a415-47c5fc688cf8-extensions-socket-volume\") pod \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " Nov 03 20:15:49.458175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.457989 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd08483-1bd3-46f7-a415-47c5fc688cf8-kube-api-access-v9545\" (UniqueName: \"kubernetes.io/projected/8cd08483-1bd3-46f7-a415-47c5fc688cf8-kube-api-access-v9545\") pod \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\" (UID: \"8cd08483-1bd3-46f7-a415-47c5fc688cf8\") " Nov 03 20:15:49.458357 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.458332 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd08483-1bd3-46f7-a415-47c5fc688cf8-extensions-socket-volume" pod "8cd08483-1bd3-46f7-a415-47c5fc688cf8" (UID: "8cd08483-1bd3-46f7-a415-47c5fc688cf8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:49.460214 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.460180 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd08483-1bd3-46f7-a415-47c5fc688cf8-kube-api-access-v9545" pod "8cd08483-1bd3-46f7-a415-47c5fc688cf8" (UID: "8cd08483-1bd3-46f7-a415-47c5fc688cf8"). InnerVolumeSpecName "kube-api-access-v9545". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:49.559165 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.559073 4314 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8cd08483-1bd3-46f7-a415-47c5fc688cf8-extensions-socket-volume\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:49.559165 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.559108 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9545\" (UniqueName: \"kubernetes.io/projected/8cd08483-1bd3-46f7-a415-47c5fc688cf8-kube-api-access-v9545\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:49.811625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.811512 4314 generic.go:356] "Generic (PLEG): container finished" podID="8cd08483-1bd3-46f7-a415-47c5fc688cf8" containerID="90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d" exitCode=0 Nov 03 20:15:49.811625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.811561 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" Nov 03 20:15:49.811625 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.811599 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" event={"ID":"8cd08483-1bd3-46f7-a415-47c5fc688cf8","Type":"ContainerDied","Data":"90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d"} Nov 03 20:15:49.811956 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.811662 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd" event={"ID":"8cd08483-1bd3-46f7-a415-47c5fc688cf8","Type":"ContainerDied","Data":"dbfba45e7fe8e2aa3aceb115f61b071a7e6c0f9cf780dee1af99d64b69e5b556"} Nov 03 20:15:49.811956 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.811687 4314 scope.go:122] "RemoveContainer" containerID="90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d" Nov 03 20:15:49.824130 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.824113 4314 scope.go:122] "RemoveContainer" containerID="90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d" Nov 03 20:15:49.824452 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:15:49.824431 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d\": container with ID starting with 90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d not found: ID does not exist" containerID="90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d" Nov 03 20:15:49.824536 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.824463 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d"} err="failed to get container status \"90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d\": rpc error: code = NotFound desc = could not find container \"90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d\": container with ID starting with 90c8aaeb7c40372853910c2cb839789566cc3b0b0f8ea2c8cab27c686755792d not found: ID does not exist" Nov 03 20:15:49.834498 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.834474 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd"] Nov 03 20:15:49.840909 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:49.840864 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8bntd"] Nov 03 20:15:50.426319 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:15:50.426278 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd08483-1bd3-46f7-a415-47c5fc688cf8" path="/var/lib/kubelet/pods/8cd08483-1bd3-46f7-a415-47c5fc688cf8/volumes" Nov 03 20:15:51.990014 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:15:51.989974 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:16:21.989886 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:16:21.989843 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:16:33.230980 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:33.230944 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:16:33.246431 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:33.246402 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:16:50.070180 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.070122 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd"] Nov 03 20:16:50.070757 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.070735 4314 cpu_manager.go:414] "containerMap: removing container" podUID="1e0a6767-efec-4bee-bbfc-63a8a0f508e8" containerName="revoke-keys" Nov 03 20:16:50.070813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.070758 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0a6767-efec-4bee-bbfc-63a8a0f508e8" containerName="revoke-keys" Nov 03 20:16:50.070813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.070802 4314 cpu_manager.go:414] "containerMap: removing container" podUID="8cd08483-1bd3-46f7-a415-47c5fc688cf8" containerName="manager" Nov 03 20:16:50.070813 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.070809 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd08483-1bd3-46f7-a415-47c5fc688cf8" containerName="manager" Nov 03 20:16:50.070955 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.070908 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0a6767-efec-4bee-bbfc-63a8a0f508e8" containerName="revoke-keys" Nov 03 20:16:50.070955 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.070927 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd08483-1bd3-46f7-a415-47c5fc688cf8" containerName="manager" Nov 03 20:16:50.075631 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.075610 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.084539 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.084041 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd"] Nov 03 20:16:50.085139 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.085117 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z646t\"" Nov 03 20:16:50.147953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.147901 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/13d71d64-7040-4c18-a294-4beaf4d86365-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vnhpd\" (UID: \"13d71d64-7040-4c18-a294-4beaf4d86365\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.148155 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.148040 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflkh\" (UniqueName: \"kubernetes.io/projected/13d71d64-7040-4c18-a294-4beaf4d86365-kube-api-access-lflkh\") pod \"kuadrant-operator-controller-manager-678d87766f-vnhpd\" (UID: \"13d71d64-7040-4c18-a294-4beaf4d86365\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.249426 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.249379 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/13d71d64-7040-4c18-a294-4beaf4d86365-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vnhpd\" (UID: \"13d71d64-7040-4c18-a294-4beaf4d86365\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.249655 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.249452 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflkh\" (UniqueName: \"kubernetes.io/projected/13d71d64-7040-4c18-a294-4beaf4d86365-kube-api-access-lflkh\") pod \"kuadrant-operator-controller-manager-678d87766f-vnhpd\" (UID: \"13d71d64-7040-4c18-a294-4beaf4d86365\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.249914 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.249887 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/13d71d64-7040-4c18-a294-4beaf4d86365-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vnhpd\" (UID: \"13d71d64-7040-4c18-a294-4beaf4d86365\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.258279 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.258243 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflkh\" (UniqueName: \"kubernetes.io/projected/13d71d64-7040-4c18-a294-4beaf4d86365-kube-api-access-lflkh\") pod \"kuadrant-operator-controller-manager-678d87766f-vnhpd\" (UID: \"13d71d64-7040-4c18-a294-4beaf4d86365\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.392333 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.392244 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:50.674673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:50.674536 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd"] Nov 03 20:16:50.679592 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:16:50.679545 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d71d64_7040_4c18_a294_4beaf4d86365.slice/crio-683bc6ff78f8539aa754590f2c4cda57a9666ddcc1f98dc04daa0e2d371e9ebf WatchSource:0}: Error finding container 683bc6ff78f8539aa754590f2c4cda57a9666ddcc1f98dc04daa0e2d371e9ebf: Status 404 returned error can't find the container with id 683bc6ff78f8539aa754590f2c4cda57a9666ddcc1f98dc04daa0e2d371e9ebf Nov 03 20:16:51.134843 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:51.134804 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" event={"ID":"13d71d64-7040-4c18-a294-4beaf4d86365","Type":"ContainerStarted","Data":"975bd86923280bfe509a686bf0413f4a5f7deab1c6c3c51b9b7ce00cbc410168"} Nov 03 20:16:51.135243 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:51.134861 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" event={"ID":"13d71d64-7040-4c18-a294-4beaf4d86365","Type":"ContainerStarted","Data":"683bc6ff78f8539aa754590f2c4cda57a9666ddcc1f98dc04daa0e2d371e9ebf"} Nov 03 20:16:51.135243 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:51.134883 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:16:51.157767 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:16:51.157715 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" podStartSLOduration=1.157701179 podStartE2EDuration="1.157701179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:51.154883313 +0000 UTC m=+2718.859722653" watchObservedRunningTime="2026-11-03 20:16:51.157701179 +0000 UTC m=+2718.862540518" Nov 03 20:16:51.989728 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:16:51.989689 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:17:02.142099 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:17:02.142064 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnhpd" Nov 03 20:17:07.520874 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:17:07.520826 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:17:21.990500 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:17:21.990459 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:17:22.459175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:17:22.459134 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:17:27.060617 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:17:27.059886 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:17:41.242930 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:17:41.242893 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:17:49.547125 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:17:49.547081 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:17:51.990799 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:17:51.990761 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:18:03.441815 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:18:03.441777 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:18:21.222490 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:18:21.222446 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:18:21.991595 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:18:21.991537 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:18:43.947941 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:18:43.947906 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:18:51.993483 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:18:51.993381 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:18:56.522549 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:18:56.522513 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:19:06.954881 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:19:06.954837 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:19:21.989888 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:19:21.989848 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:19:26.738356 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:19:26.738313 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:19:39.449738 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:19:39.449700 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:19:51.142970 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:19:51.142925 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:19:52.006370 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:19:52.006323 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:19:56.846058 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:19:56.845996 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:20:03.508891 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:03.508853 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:20:07.023732 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:07.023689 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc"] Nov 03 20:20:07.025398 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:07.025363 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6338e751-n5ljc"] Nov 03 20:20:07.836105 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:07.836061 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:20:08.439142 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:08.439108 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40991a6-ace3-4956-9dd7-42c1c6f80ff8" path="/var/lib/kubelet/pods/b40991a6-ace3-4956-9dd7-42c1c6f80ff8/volumes" Nov 03 20:20:21.996777 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:20:21.996729 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:20:23.734546 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:23.734509 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:20:32.389399 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.389361 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6"] Nov 03 20:20:32.393826 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.393792 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.402690 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.402648 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6"] Nov 03 20:20:32.490745 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.490703 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfe9f61-7241-40db-983a-eb0c39182902-maas-api-tls\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.490932 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.490762 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfe9f61-7241-40db-983a-eb0c39182902-metrics-certs\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.491024 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.490997 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfe9f61-7241-40db-983a-eb0c39182902-openshift-service-ca\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.491101 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.491086 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q74qv\" (UniqueName: \"kubernetes.io/projected/cbfe9f61-7241-40db-983a-eb0c39182902-kube-api-access-q74qv\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.591686 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.591642 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q74qv\" (UniqueName: \"kubernetes.io/projected/cbfe9f61-7241-40db-983a-eb0c39182902-kube-api-access-q74qv\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.591883 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.591708 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfe9f61-7241-40db-983a-eb0c39182902-maas-api-tls\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.591883 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.591735 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfe9f61-7241-40db-983a-eb0c39182902-metrics-certs\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.591883 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.591822 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfe9f61-7241-40db-983a-eb0c39182902-openshift-service-ca\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.592633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.592567 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfe9f61-7241-40db-983a-eb0c39182902-openshift-service-ca\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.594412 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.594386 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfe9f61-7241-40db-983a-eb0c39182902-maas-api-tls\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.594508 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.594433 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfe9f61-7241-40db-983a-eb0c39182902-metrics-certs\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.599729 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.599697 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q74qv\" (UniqueName: \"kubernetes.io/projected/cbfe9f61-7241-40db-983a-eb0c39182902-kube-api-access-q74qv\") pod \"maas-api-7d9ffb8588-vvvx6\" (UID: \"cbfe9f61-7241-40db-983a-eb0c39182902\") " pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:32.716726 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:32.716622 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:33.004859 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:33.004831 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6"] Nov 03 20:20:33.007112 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:20:33.007083 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfe9f61_7241_40db_983a_eb0c39182902.slice/crio-3f8e721788de2315189d55f1566f670b274ca78153ab24d6f78ef5fb9ffdb9bc WatchSource:0}: Error finding container 3f8e721788de2315189d55f1566f670b274ca78153ab24d6f78ef5fb9ffdb9bc: Status 404 returned error can't find the container with id 3f8e721788de2315189d55f1566f670b274ca78153ab24d6f78ef5fb9ffdb9bc Nov 03 20:20:33.008533 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:33.008513 4314 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:20:33.365047 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:33.364964 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" event={"ID":"cbfe9f61-7241-40db-983a-eb0c39182902","Type":"ContainerStarted","Data":"3f8e721788de2315189d55f1566f670b274ca78153ab24d6f78ef5fb9ffdb9bc"} Nov 03 20:20:34.372414 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:34.372373 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" event={"ID":"cbfe9f61-7241-40db-983a-eb0c39182902","Type":"ContainerStarted","Data":"889d9f09b834779a511e772f9b7f2fc7ce8616050b322af948fc2e471e261295"} Nov 03 20:20:34.372918 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:34.372532 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:34.389723 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:34.389659 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" podStartSLOduration=1.987128813 podStartE2EDuration="2.389641515s" totalImagesPullingTime="402.512702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:34.388923971 +0000 UTC m=+2942.093763310" watchObservedRunningTime="2026-11-03 20:20:34.389641515 +0000 UTC m=+2942.094480858" Nov 03 20:20:38.341637 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:38.341599 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:20:40.383247 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:40.383195 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d9ffb8588-vvvx6" Nov 03 20:20:40.431014 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:40.430976 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl"] Nov 03 20:20:40.431252 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:40.431227 4314 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" podUID="cbf7efc4-af20-4a85-90d6-e1ee635f9be1" containerName="maas-api" containerID="cri-o://b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6" gracePeriod=30 Nov 03 20:20:41.695732 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.695703 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 20:20:41.783984 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.783944 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-openshift-service-ca\") pod \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " Nov 03 20:20:41.784208 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.784025 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-maas-api-tls\") pod \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " Nov 03 20:20:41.784208 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.784071 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-kube-api-access-5lvsc\" (UniqueName: \"kubernetes.io/projected/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-kube-api-access-5lvsc\") pod \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " Nov 03 20:20:41.784208 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.784160 4314 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-metrics-certs\") pod \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\" (UID: \"cbf7efc4-af20-4a85-90d6-e1ee635f9be1\") " Nov 03 20:20:41.784420 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.784389 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-openshift-service-ca" pod "cbf7efc4-af20-4a85-90d6-e1ee635f9be1" (UID: "cbf7efc4-af20-4a85-90d6-e1ee635f9be1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:41.784565 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.784545 4314 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-openshift-service-ca\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:41.786642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.786608 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-kube-api-access-5lvsc" pod "cbf7efc4-af20-4a85-90d6-e1ee635f9be1" (UID: "cbf7efc4-af20-4a85-90d6-e1ee635f9be1"). InnerVolumeSpecName "kube-api-access-5lvsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:41.786642 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.786629 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-metrics-certs" pod "cbf7efc4-af20-4a85-90d6-e1ee635f9be1" (UID: "cbf7efc4-af20-4a85-90d6-e1ee635f9be1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:41.787142 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.787123 4314 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-maas-api-tls" pod "cbf7efc4-af20-4a85-90d6-e1ee635f9be1" (UID: "cbf7efc4-af20-4a85-90d6-e1ee635f9be1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:41.885281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.885246 4314 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-metrics-certs\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:41.885281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.885279 4314 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-maas-api-tls\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:41.885281 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:41.885290 4314 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvsc\" (UniqueName: \"kubernetes.io/projected/cbf7efc4-af20-4a85-90d6-e1ee635f9be1-kube-api-access-5lvsc\") on node \"ip-10-0-133-204.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:42.414915 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.414881 4314 generic.go:356] "Generic (PLEG): container finished" podID="cbf7efc4-af20-4a85-90d6-e1ee635f9be1" containerID="b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6" exitCode=0 Nov 03 20:20:42.415136 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.414931 4314 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" Nov 03 20:20:42.415136 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.414966 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" event={"ID":"cbf7efc4-af20-4a85-90d6-e1ee635f9be1","Type":"ContainerDied","Data":"b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6"} Nov 03 20:20:42.415136 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.415028 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl" event={"ID":"cbf7efc4-af20-4a85-90d6-e1ee635f9be1","Type":"ContainerDied","Data":"02491e991d6d151a96d3b0f57ed8c243739c3cc905be3402e2b59808646ffd2a"} Nov 03 20:20:42.415136 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.415061 4314 scope.go:122] "RemoveContainer" containerID="b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6" Nov 03 20:20:42.426368 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.426347 4314 scope.go:122] "RemoveContainer" containerID="b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6" Nov 03 20:20:42.426708 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:20:42.426679 4314 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6\": container with ID starting with b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6 not found: ID does not exist" containerID="b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6" Nov 03 20:20:42.426817 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.426718 4314 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6"} err="failed to get container status \"b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6\": rpc error: code = NotFound desc = could not find container \"b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6\": container with ID starting with b6db84d2ba664b332fc8845bcb3679746e207c1e0e9689aeac434dd6b2c64ef6 not found: ID does not exist" Nov 03 20:20:42.437242 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.437209 4314 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl"] Nov 03 20:20:42.440712 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:42.440687 4314 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6599bcbcd4-7z8xl"] Nov 03 20:20:44.427256 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:20:44.427224 4314 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf7efc4-af20-4a85-90d6-e1ee635f9be1" path="/var/lib/kubelet/pods/cbf7efc4-af20-4a85-90d6-e1ee635f9be1/volumes" Nov 03 20:20:51.989565 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:20:51.989521 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:21:02.987565 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:21:02.987523 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:21:15.009557 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:21:15.009518 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:21:21.990261 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:21:21.990220 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:21:33.260546 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:21:33.260518 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:21:33.276924 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:21:33.276898 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:21:38.405376 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:21:38.405340 4314 scope.go:122] "RemoveContainer" containerID="1c07e01f0f9ae1cb5ae0601d5632a483302cdc86a4a186758659808d8224dd45" Nov 03 20:21:38.413784 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:21:38.413744 4314 scope.go:122] "RemoveContainer" containerID="5bc48cdde80b380f008d5d9cee724812c57fb0444afe00c2b52e2f6bf65aeef0" Nov 03 20:21:51.990439 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:21:51.990351 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:22:21.990002 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:22:21.989955 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:22:51.990390 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:22:51.990345 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:23:07.445001 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:23:07.444964 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:23:14.706986 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:23:14.706950 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:23:19.747841 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:23:19.747807 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:23:22.050367 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:23:22.050328 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:23:35.014673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:23:35.014634 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:23:51.989332 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:23:51.989289 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:24:01.615992 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:01.615951 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-n7khn"] Nov 03 20:24:03.619789 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:03.619741 4314 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:24:03.620490 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:03.620452 4314 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-lx2rx" podUID="361708af-2ed1-40e2-8da6-a3e28562ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:17.797298 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:17.797260 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xp8mn_89d72820-7b45-41bd-8252-b25c4c1ef048/manager/0.log" Nov 03 20:24:18.002975 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:18.002930 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6mwnz_900bc5dd-60bc-4597-9d8d-33c2551b5213/proxy/0.log" Nov 03 20:24:18.883656 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:18.883627 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d9ffb8588-vvvx6_cbfe9f61-7241-40db-983a-eb0c39182902/maas-api/0.log" Nov 03 20:24:21.989322 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:24:21.989287 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:24:22.080304 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:22.080274 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ffrlm_93fcd6a5-399a-4300-ad54-afdd837f808a/kuadrant-console-plugin/0.log" Nov 03 20:24:22.295048 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:22.295007 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vnhpd_13d71d64-7040-4c18-a294-4beaf4d86365/manager/0.log" Nov 03 20:24:22.500537 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:22.500506 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n7khn_81469cc9-29b4-4679-bc25-acacb15bd34d/limitador/0.log" Nov 03 20:24:23.872392 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:23.872361 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-dmhc8_f1ee53b4-83e7-4974-807c-f89e3df209d9/kube-auth-proxy/0.log" Nov 03 20:24:24.577848 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:24.577816 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-h8wkg_951574bf-7a44-46a7-bb31-aa33bdd6792d/istio-proxy/0.log" Nov 03 20:24:24.896781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:24.896694 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lwvfc_c4042181-5019-4886-9677-c736a3a2c48d/istio-proxy/0.log" Nov 03 20:24:25.065781 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:25.065752 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lx2rx_361708af-2ed1-40e2-8da6-a3e28562ce7b/istio-proxy/0.log" Nov 03 20:24:25.936487 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:25.936391 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-76c4k_1205a003-d0a7-4aa0-8cdd-b38e0afd4e63/payload-pre-processing/0.log" Nov 03 20:24:26.105464 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:26.105439 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:24:27.216653 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:27.216615 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl_0067d07b-adb9-4976-a4fb-6bdcec9091cf/main/0.log" Nov 03 20:24:27.552538 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:27.552498 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-sxc97_da4973d4-3de9-4f1e-96cb-2def28c1396f/main/0.log" Nov 03 20:24:27.887550 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:27.887448 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95_3e2993e6-3c87-42aa-9279-fc3d11d9e1c2/main/0.log" Nov 03 20:24:28.223820 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:28.223725 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m_6ab1ff97-e41f-4a1b-900e-768abaf58008/main/0.log" Nov 03 20:24:38.832145 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:38.832112 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-xp8mn_89d72820-7b45-41bd-8252-b25c4c1ef048/manager/0.log" Nov 03 20:24:39.037294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:39.037263 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6mwnz_900bc5dd-60bc-4597-9d8d-33c2551b5213/proxy/0.log" Nov 03 20:24:39.700633 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:39.700565 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d9ffb8588-vvvx6_cbfe9f61-7241-40db-983a-eb0c39182902/maas-api/0.log" Nov 03 20:24:42.444518 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:42.444489 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ffrlm_93fcd6a5-399a-4300-ad54-afdd837f808a/kuadrant-console-plugin/0.log" Nov 03 20:24:42.621000 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:42.620963 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vnhpd_13d71d64-7040-4c18-a294-4beaf4d86365/manager/0.log" Nov 03 20:24:42.796848 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:42.796814 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n7khn_81469cc9-29b4-4679-bc25-acacb15bd34d/limitador/0.log" Nov 03 20:24:43.974220 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:43.974191 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-dmhc8_f1ee53b4-83e7-4974-807c-f89e3df209d9/kube-auth-proxy/0.log" Nov 03 20:24:44.641350 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:44.641317 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-h8wkg_951574bf-7a44-46a7-bb31-aa33bdd6792d/istio-proxy/0.log" Nov 03 20:24:44.990690 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:44.990554 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lwvfc_c4042181-5019-4886-9677-c736a3a2c48d/istio-proxy/0.log" Nov 03 20:24:45.156755 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:45.156718 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lx2rx_361708af-2ed1-40e2-8da6-a3e28562ce7b/istio-proxy/0.log" Nov 03 20:24:46.005738 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:46.005710 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-76c4k_1205a003-d0a7-4aa0-8cdd-b38e0afd4e63/payload-pre-processing/0.log" Nov 03 20:24:46.170002 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:46.169967 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:24:46.820436 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:46.820403 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5zwgl_0067d07b-adb9-4976-a4fb-6bdcec9091cf/main/0.log" Nov 03 20:24:47.138090 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:47.138004 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-sxc97_da4973d4-3de9-4f1e-96cb-2def28c1396f/main/0.log" Nov 03 20:24:47.476489 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:47.476398 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hlg95_3e2993e6-3c87-42aa-9279-fc3d11d9e1c2/main/0.log" Nov 03 20:24:47.797599 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:24:47.797553 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4zh2m_6ab1ff97-e41f-4a1b-900e-768abaf58008/main/0.log" Nov 03 20:24:51.989577 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:24:51.989534 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:25:21.989037 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:25:21.988994 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:25:23.497133 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:25:23.497104 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:25:23.665206 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:25:23.665167 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:25:51.990871 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:25:51.990753 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:26:06.630987 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:06.630958 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:26:06.786371 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:06.786342 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:26:10.379512 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:10.379475 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8z8g9_7bac36d0-ca54-4a9c-83cd-c58ba9f58c37/global-pull-secret-syncer/0.log" Nov 03 20:26:10.688814 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:10.688724 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8cmvr_2d56c5c2-1d76-490b-b853-3e8d15662fa5/konnectivity-agent/0.log" Nov 03 20:26:10.790490 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:10.790450 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-204.us-east-2.compute.internal_70af56b747e8dbbb62b5b41a5fc7a4ec/haproxy/0.log" Nov 03 20:26:18.309020 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:18.308995 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ffrlm_93fcd6a5-399a-4300-ad54-afdd837f808a/kuadrant-console-plugin/0.log" Nov 03 20:26:18.387918 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:18.387877 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vnhpd_13d71d64-7040-4c18-a294-4beaf4d86365/manager/0.log" Nov 03 20:26:18.422020 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:18.421983 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-n7khn_81469cc9-29b4-4679-bc25-acacb15bd34d/limitador/0.log" Nov 03 20:26:19.994640 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:19.994598 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w2l2f_5c974e75-4f65-4ac9-913f-74b063a48dc1/kube-state-metrics/0.log" Nov 03 20:26:20.015508 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.015470 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w2l2f_5c974e75-4f65-4ac9-913f-74b063a48dc1/kube-rbac-proxy-main/0.log" Nov 03 20:26:20.042796 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.042764 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-w2l2f_5c974e75-4f65-4ac9-913f-74b063a48dc1/kube-rbac-proxy-self/0.log" Nov 03 20:26:20.112135 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.112104 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-fv589_3b6e1648-d86b-4931-942a-0134faa484ae/monitoring-plugin/0.log" Nov 03 20:26:20.121026 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.120993 4314 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9"] Nov 03 20:26:20.121595 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.121565 4314 cpu_manager.go:414] "containerMap: removing container" podUID="cbf7efc4-af20-4a85-90d6-e1ee635f9be1" containerName="maas-api" Nov 03 20:26:20.121670 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.121601 4314 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf7efc4-af20-4a85-90d6-e1ee635f9be1" containerName="maas-api" Nov 03 20:26:20.121839 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.121811 4314 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf7efc4-af20-4a85-90d6-e1ee635f9be1" containerName="maas-api" Nov 03 20:26:20.125378 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.125356 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.130563 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.130532 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9"] Nov 03 20:26:20.133010 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.132986 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fg8c7\"/\"openshift-service-ca.crt\"" Nov 03 20:26:20.133175 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.133000 4314 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fg8c7\"/\"kube-root-ca.crt\"" Nov 03 20:26:20.133253 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.133008 4314 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fg8c7\"/\"default-dockercfg-8dkgq\"" Nov 03 20:26:20.149292 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.149253 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-proc\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.149482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.149331 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-sys\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.149482 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.149378 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswdv\" (UniqueName: \"kubernetes.io/projected/a0bd42d1-cdaa-4341-8448-be4a946ca31f-kube-api-access-qswdv\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.149649 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.149499 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-podres\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.149649 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.149558 4314 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-lib-modules\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251113 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251015 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-lib-modules\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251113 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251068 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-proc\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251113 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251112 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-sys\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251141 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswdv\" (UniqueName: \"kubernetes.io/projected/a0bd42d1-cdaa-4341-8448-be4a946ca31f-kube-api-access-qswdv\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251176 4314 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-podres\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251181 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-lib-modules\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251217 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-proc\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251217 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-sys\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.251372 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.251272 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0bd42d1-cdaa-4341-8448-be4a946ca31f-podres\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.259129 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.259094 4314 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswdv\" (UniqueName: \"kubernetes.io/projected/a0bd42d1-cdaa-4341-8448-be4a946ca31f-kube-api-access-qswdv\") pod \"perf-node-gather-daemonset-2xgw9\" (UID: \"a0bd42d1-cdaa-4341-8448-be4a946ca31f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.277386 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.277359 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwnmp_8cd36336-cf9e-46f7-9e42-f299f8ef0e26/node-exporter/0.log" Nov 03 20:26:20.314793 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.314760 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwnmp_8cd36336-cf9e-46f7-9e42-f299f8ef0e26/kube-rbac-proxy/0.log" Nov 03 20:26:20.356548 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.356520 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwnmp_8cd36336-cf9e-46f7-9e42-f299f8ef0e26/init-textfile/0.log" Nov 03 20:26:20.442515 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.442476 4314 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:20.543294 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.543219 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wl2v6_ea79c75d-c7c8-452f-b84e-ec8d94449400/kube-rbac-proxy-main/0.log" Nov 03 20:26:20.594132 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.594098 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wl2v6_ea79c75d-c7c8-452f-b84e-ec8d94449400/kube-rbac-proxy-self/0.log" Nov 03 20:26:20.643015 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.642983 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wl2v6_ea79c75d-c7c8-452f-b84e-ec8d94449400/openshift-state-metrics/0.log" Nov 03 20:26:20.719561 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.719524 4314 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9"] Nov 03 20:26:20.719979 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.719958 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e86509fa-ee71-4083-8943-491c49eebcfe/prometheus/0.log" Nov 03 20:26:20.723973 ip-10-0-133-204 kubenswrapper[4314]: W1103 20:26:20.723943 4314 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda0bd42d1_cdaa_4341_8448_be4a946ca31f.slice/crio-e3af292dbc195b17f47c43b761271f159b0a50998930a0b3acf509b6343240b4 WatchSource:0}: Error finding container e3af292dbc195b17f47c43b761271f159b0a50998930a0b3acf509b6343240b4: Status 404 returned error can't find the container with id e3af292dbc195b17f47c43b761271f159b0a50998930a0b3acf509b6343240b4 Nov 03 20:26:20.772775 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.772747 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e86509fa-ee71-4083-8943-491c49eebcfe/config-reloader/0.log" Nov 03 20:26:20.802635 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.802538 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e86509fa-ee71-4083-8943-491c49eebcfe/thanos-sidecar/0.log" Nov 03 20:26:20.834460 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.834429 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e86509fa-ee71-4083-8943-491c49eebcfe/kube-rbac-proxy-web/0.log" Nov 03 20:26:20.865406 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.865376 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e86509fa-ee71-4083-8943-491c49eebcfe/kube-rbac-proxy/0.log" Nov 03 20:26:20.893597 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.893549 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e86509fa-ee71-4083-8943-491c49eebcfe/kube-rbac-proxy-thanos/0.log" Nov 03 20:26:20.922467 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:20.922439 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e86509fa-ee71-4083-8943-491c49eebcfe/init-config-reloader/0.log" Nov 03 20:26:21.016153 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:21.016123 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-cjcmt_4ab98eaf-1317-4f84-aab7-b3a89831ea07/prometheus-operator-admission-webhook/0.log" Nov 03 20:26:21.117273 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:21.117182 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" event={"ID":"a0bd42d1-cdaa-4341-8448-be4a946ca31f","Type":"ContainerStarted","Data":"624853edf747dddd6a04b4c95988536c75c001e2686f5da2d573aa5b0dfbc316"} Nov 03 20:26:21.117273 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:21.117235 4314 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" event={"ID":"a0bd42d1-cdaa-4341-8448-be4a946ca31f","Type":"ContainerStarted","Data":"e3af292dbc195b17f47c43b761271f159b0a50998930a0b3acf509b6343240b4"} Nov 03 20:26:21.117273 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:21.117265 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:21.133833 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:21.133772 4314 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" podStartSLOduration=1.133757159 podStartE2EDuration="1.133757159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:21.132853732 +0000 UTC m=+3288.837693074" watchObservedRunningTime="2026-11-03 20:26:21.133757159 +0000 UTC m=+3288.838596499" Nov 03 20:26:21.990434 ip-10-0-133-204 kubenswrapper[4314]: E1103 20:26:21.990392 4314 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6gx66" containerName="extractor" Nov 03 20:26:23.630654 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:23.630614 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-28czb_b4cf18aa-f70f-4b75-ab34-c85cadb4c7ab/download-server/0.log" Nov 03 20:26:25.464029 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:25.463995 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlqmb_e287d664-857c-4fb1-beb0-f248ef548609/dns/0.log" Nov 03 20:26:25.488924 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:25.488893 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlqmb_e287d664-857c-4fb1-beb0-f248ef548609/kube-rbac-proxy/0.log" Nov 03 20:26:25.638425 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:25.638391 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ns5zn_0225e321-66c4-4269-81d4-051793673f4d/dns-node-resolver/0.log" Nov 03 20:26:26.371925 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:26.371883 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57899686b7-qh8vj_f2d834ce-9da4-4dfe-8433-89df929dd609/registry/0.log" Nov 03 20:26:26.466943 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:26.466904 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zj9jh_3fbf00d3-e30f-4399-b770-6545f69f15ce/node-ca/0.log" Nov 03 20:26:27.134041 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:27.134008 4314 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-2xgw9" Nov 03 20:26:28.917454 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:28.917418 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-dmhc8_f1ee53b4-83e7-4974-807c-f89e3df209d9/kube-auth-proxy/0.log" Nov 03 20:26:29.061976 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:29.061946 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-h8wkg_951574bf-7a44-46a7-bb31-aa33bdd6792d/istio-proxy/0.log" Nov 03 20:26:29.131039 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:29.130992 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lwvfc_c4042181-5019-4886-9677-c736a3a2c48d/istio-proxy/0.log" Nov 03 20:26:29.165641 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:29.165603 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lx2rx_361708af-2ed1-40e2-8da6-a3e28562ce7b/istio-proxy/0.log" Nov 03 20:26:29.327431 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:29.327394 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-76c4k_1205a003-d0a7-4aa0-8cdd-b38e0afd4e63/payload-pre-processing/0.log" Nov 03 20:26:29.359101 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:29.359068 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:26:31.396526 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:31.396490 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-dmhc8_f1ee53b4-83e7-4974-807c-f89e3df209d9/kube-auth-proxy/0.log" Nov 03 20:26:31.482393 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:31.482343 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-h8wkg_951574bf-7a44-46a7-bb31-aa33bdd6792d/istio-proxy/0.log" Nov 03 20:26:31.520953 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:31.520914 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lwvfc_c4042181-5019-4886-9677-c736a3a2c48d/istio-proxy/0.log" Nov 03 20:26:31.541673 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:31.541635 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lx2rx_361708af-2ed1-40e2-8da6-a3e28562ce7b/istio-proxy/0.log" Nov 03 20:26:31.643541 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:31.643498 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-76c4k_1205a003-d0a7-4aa0-8cdd-b38e0afd4e63/payload-pre-processing/0.log" Nov 03 20:26:31.660677 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:31.660598 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:26:31.904373 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:31.904332 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c6c8z_9fd524a3-fa08-43aa-8db1-71bb34dd961a/serve-healthcheck-canary/0.log" Nov 03 20:26:32.727424 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:32.727390 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6gx66_62450e22-1ab9-4663-8960-cceab528bd29/kube-rbac-proxy/0.log" Nov 03 20:26:32.774577 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:32.774549 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6gx66_62450e22-1ab9-4663-8960-cceab528bd29/exporter/0.log" Nov 03 20:26:32.816622 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:32.816591 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6gx66_62450e22-1ab9-4663-8960-cceab528bd29/extractor/0.log" Nov 03 20:26:33.290154 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:33.290126 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:26:33.315693 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:33.315668 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xss4s_c8b6db40-add7-4eca-9480-e174e021fdc7/ovn-acl-logging/0.log" Nov 03 20:26:35.560242 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.560209 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f46798c78-dmhc8_f1ee53b4-83e7-4974-807c-f89e3df209d9/kube-auth-proxy/0.log" Nov 03 20:26:35.639421 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.639387 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-h8wkg_951574bf-7a44-46a7-bb31-aa33bdd6792d/istio-proxy/0.log" Nov 03 20:26:35.680227 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.680190 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lwvfc_c4042181-5019-4886-9677-c736a3a2c48d/istio-proxy/0.log" Nov 03 20:26:35.701951 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.701917 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-lx2rx_361708af-2ed1-40e2-8da6-a3e28562ce7b/istio-proxy/0.log" Nov 03 20:26:35.740877 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.740845 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-bd9746b54-5blw5_e0989ad6-060e-4da2-a646-6095e895ad09/manager/0.log" Nov 03 20:26:35.769419 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.769394 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-bd9746b54-5blw5_e0989ad6-060e-4da2-a646-6095e895ad09/copy-manifests/0.log" Nov 03 20:26:35.794329 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.794295 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-76c4k_1205a003-d0a7-4aa0-8cdd-b38e0afd4e63/payload-pre-processing/0.log" Nov 03 20:26:35.812461 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:35.812381 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-j9kp5_501a6ce3-bb3e-42ac-84cf-7bb98fb19d5a/payload-processing/0.log" Nov 03 20:26:37.167628 ip-10-0-133-204 kubenswrapper[4314]: I1103 20:26:37.167559 4314 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-szdd6_bd09c84b-16b6-4b4d-8a6f-ad92d9a1233f/jobset-operator/0.log"